Use nebula_dart_gdbc to play with graph databases on mobile devices, so cool!

nebula_dart_gdbc is a Dart language client for accessing NebulaGraph, developed under the specifications of dart_gdbc.

dart_gdbc is a set of standard data interfaces for graph databases defined using the Dart language. The overall idea refers to the JDBC specification. Support for NebulaGraph has been implemented.

In the previous process of writing GraphDesk, the nebula-java (Socket) + Spring Boot (HTTP) method was used to implement the Flutter application’s access to NebulaGraph. This is tantamount to a car (Socket) that can travel at high speeds. When entering and leaving the house, you have to use the method of pushing (HTTP). This is hard to accept. Using Dart Socket to access graph databases is undoubtedly a good choice for Flutter applications. However, compared with other programming languages, Dart is currently relatively niche, has low demand, and is a front-end language. Looking at the existing graph database ecosystem, none of them support Dart.

As a result, dart_gdbc and nebula_dart_gdbc came into being.

With the opening of the Dart client, it has become possible for friends who own graph databases to use Dart to explore NebulaGraph graphs on mobile phones or even embedded platforms. For Flutter developers, you can try creating your own graph database and use nebula_dart_gdbc to directly connect to the database to implement your own applications. It is also a wonderful experience. Although it is more reasonable for the regular operation of the database to forward data through the server, it does not prevent everyone from trying this new method.

If all goes well, after Dart is completed, the GraphDesk I built before will have the following changes:

  • From the perspective of program startup, it will break free from the constraints of the Java environment and no longer need to wait for the startup process of the local Java service. There will be a qualitative change in the running experience;
  • From a query performance perspective, it will also be smoother;
  • When conditions permit, an App version will be launched in the future.

That’s it for the meaningful content. Let’s talk about the birth process of nebula_dart_gdbc. The following is very technical and non-industry people can skip it optionally.

Design and implementation of nebula_dart_gdbc

The general implementation idea of nebula_dart_gdbc is the same as that of clients in other languages. The following steps are followed:

  1. Generate NebulaGraph data access interface through thrift file;
  2. Combine the generated data access interface with Thrift to complete access to NebulaGraph;
  3. Encapsulate database connections.

Obviously, real development is not that simple. In the actual process, I encountered many problems and made many attempts, and finally came up with the current nebula_dart_gdbc. Let’s record the steps to develop a Dart client and the solutions to related problems.

Generate NebulaGraph data access interface

The first step is to download the thrift file of the data interface. This is the link given by NebulaGraph: https://github.com/vesoft-inc/nebula/tree/master/src/interface, which contains the following files:

  • common.thrift
  • graph.thrift
  • meta.thrift
  • storage.thrift

Note: After downloading, you need to add the corresponding package name to the file, such as: namespace dart nebula.graph

In the second step, after downloading the NebulaGraph thrift file required for the interface, we download Thrift again. This is the official download address of Thrift: https://thrift.apache.org/download. The latest version 0.18.1 is used here ( If 0.18.1 is not the latest version when you read this article, just download the latest version).

The third step is to generate Dart code file by file. Just use the following command:

thrift --gen dart common.thrift
thrift --gen dart graph.thrift
thrift --gen dart meta.thrift
thrift --gen dart storage.thrift

During this process, we will encounter some problems, and their solutions are recorded here.

  • Issue 1: Files encounter “non-language neutral markup compilation” issues;
    • Solution: Click the error message to delete the language tag in the file.
  • Question 2: The Dart version used here is 2.19.0. The Dart code generated by thrift-0.18.1 is not compatible with the current version syntax, and the versions are very different. The error reported is as shown in the figure below. Of course, this is only a part of it. There are about 300 files that actually reported errors.
  • (This is a picture of retail investors cheering and programmers crying)
    • Solution: Change the files one by one according to the error reported by the editor. Probably the stupidest way, took several hours. A better way would be to let thrift’s compiler support the latest Dart version, but this approach may take more time and may not be successful. (The drums for retreating are loud and they retreat…)

The fourth step is to put the generated Dart code into the project.

At this point, “generating the NebulaGraph data access interface through the thrift file” is completed.

Data access interface combined with Thrift

Here, let’s insert an advertisement first (please read it patiently).

Let me introduce to you a barbecue stall. The characteristic of this store is that one waiter has one tray, and this waiter only serves one table. The waiter solemnly used a tray and brought a blank piece of paper to the customers, and the customers took turns writing down their favorite beef skewers, roasted gluten, roasted cauliflower, etc., and then selected the seasonings such as black pepper, cumin, etc. After placing the paper on the tray, the waiter will send it to the barbecue master. The barbecue master will follow the instructions on the paper and put the dishes on the plate after baking and place them on the original tray. The waiter then brings it to the customers, and the customers take away the dishes they ordered.
The name of this barbecue stall is: Thrift.

This might be the strangest Thrift trope yet. Thrift is an RPC framework that converts interface parameters into binary data. Next, we introduce two important interfaces of Thrift: TProtocol and TTransport. Taking the barbecue stall just now as an example, the corresponding relationship can be roughly as follows (it may be slightly inappropriate):

  • TProtocol: There can be many different customers responsible for writing content
  • TTransport: tray + waiter, responsible for delivering content
  • TSocketTransport: waiter
  • THeaderTransport: pallet
  • Socket connected IP and port: table number
  • Output stream passed to server: paper
  • Input parameters for calling the interface: the content on the paper
  • The input stream passed to the client: dish
  • The return value of calling the interface: beef skewers, roasted gluten, roasted cauliflower…black pepper, cumin…
  • Server: Grill Master

Okay, commercial over. I believe that if you are smart, you have a certain understanding of Thrift after reading this. Let’s continue our Dart journey:

First, refer to the nebula-java source code to call GraphServiceClient.

I encountered two problems at this stage:

  • Question 1: nebula-java uses fbthrift, but fbthrift does not have Dart implementation.
    • Solution: Use Apache Thrift’s Dart implementation.
  • Question 2: There are big differences between the two implementation solutions. For example, Apache Thrift does not have THeaderTransport. What they have in common is that they both use Socket.
    • Solution: Let’s lay some groundwork here first, and then establish a Socket connection.

After solving the above two problems, we can access NebulaGraph through Dart Socket connection.

How do you say this step? It’s like another table has arrived, and now there are two tables. One table has customers and waiters who can only speak Hokkien, and one table has customers and waiters who can only speak Mandarin. The barbecue chef doesn’t care about anything, he only looks at the menu. The new table has only one request, which is to order exactly the same thing as the table next door… Since the two tables do not understand each other, if there is no accident, there will be an accident here.

Okay, we now have a few problems:

  • Problem 1: Empty data was returned and the error message was unknown.
    • Solution: First use nebula-java, set a breakpoint at the location where the Socket sends the data, copy the data, and send it directly through the Dart Socket. At this point, I realized that I had to create a debugging environment. According to my simple intuition, the client and server should be a structure similar to the reflection on the lake. So, after referring to the nebula-java client code, I decided to start a small experiment, built a server Mock, and implemented the GraphService interface by returning fake data. At this point, the server is ready for debugging.
  • Question 2: How to ensure that the structure of Mock is correct?
    • Solution: Use the nebula-java client to connect to the Mock server to see if the data can be returned normally. If the data can be returned normally, the structural accuracy of the Mock can be determined. The debugging here uses the verifyClientVersion interface which does not require many conditions.

At this point, the structure as shown in the figure is established:

  • The corresponding relationship is as follows:
    • C1:nebula-java
    • S1: NebulaGraph database
    • C2:nebula_dart_gdbc
    • S2: Mock test
  • The overall idea is as follows:
    • L1 is reliable and can send and receive accurate data;
    • L2 can send accurate data, using L2 connectivity to ensure S2 accuracy;
    • Debug C2 through L3. When the data returned by L2 and L3 are the same, it means that the data sent by C2 at this time is correct;
    • Execute L4. When the data returned by L4 and L1 are the same, it means that C2 is connected to the database through Socket.

After the basic idea was determined, a long process of translation (Java code into Dart code), debugging, translation, debugging again… began, jumping repeatedly between L2 and L3. The data representation form of each link is a pure digital array. If the length of the array or the value corresponding to the subscript is wrong, the result may fall short. Care and patience are required here. In particular, digital operations involve operations such as complement, complement, and shift. Different parameters have different byte lengths and are read and written bit by bit. Here are a few more troublesome problems during repeated horizontal jumps of L2 and L3:

  • Question 1: The Thrift version of Apache lacks many classes compared to the fbthrift version, so I won’t list them all here.
    • Solution: Translate the missing classes from fbthrift in nebula-java, and also translate the dependent third-party classes (fortunately, there are not many dependencies, only a few IO-related classes. Fortunately, the ones that do data compression The zip package has dependencies, but they are not actually used).
  • Question 2: The structure is the same, but the data sent is still different. In the Apache version, including Socket, the container used to write data is Uint8List. All numbers put in will be forcibly converted into positive numbers, but the data in fbthrift is With a plus or minus sign.
    • Solution: Modify all types to use Int8List declaration and transmission, including classes generated by thrift.
  • Question 3: There is no distinction between byte, short, int, and long in Dart. All integer types are int.
    • Solution: It is also an integer type, so you need to pay special attention to manually reduce the integer precision according to the actual situation, so as to achieve the same effect as Java in which the precision is automatically discarded when assigning values after declaring the type.

At this point, L3 has been connected, C1 and C2 accessed S2, and the data returned are consistent, and they won the mock test! Keep up the challenge:

  • Question 4: When L4 is called and connected to S1 through C2, the returned data is still empty data. It is clear that L2 and L3 get the same data from S2, which is very confusing.
    • Solution 1: Rewrite the Socket mechanism under the Apache version, abandon the use of specifying message callbacks when creating, and directly use async/await to make debugging more consistent with sequential thinking. Then, wait to read the returned results. Although the data read is still empty, this writing method is still retained in the way of using Socket.
    • Solution 2: Later, I discovered that the cause of this problem was the different writing methods between Java Socket and Dart Socket. In nebula-java, THeaderTransport is used for data transmission between interfaces. The data is sent in two parts. I am not sure whether the breakpoint is in the wrong position. What the server gets is the merged data of the two parts. The method in Dart is to splice two List into one section and send them together. The performance in S2 is exactly the same. This problem is very hidden. After a long period of time, you can’t write a line of code. Time when not working. In the end, it was changed to forcibly stripping the two List in one Dart Socket transmission and using two Streams for transmission. At this point, L4 is successful and the connectivity issue is solved. To be honest, this last bit has a touch of luck.

Encapsulating database connection

In this link, there are relatively few interruptions in the implementation of ideas, and the overall process is smooth. What takes more time is the processing of the result set.

nebula_dart_gdbc does not follow the writing method of other NebulaGraph clients. Instead, it refers to the idea of JDBC as a whole and encapsulates database connection, execution, result set, etc. to form dart_gdbc. Under this set of interface standards, nebula_dart_gdbc is implemented.

Friends who are familiar with JDBC will be familiar with the following process. The calling process of dart_gdbc is as follows:

  • Register driver
  • Get connection
  • create statement
  • Execute statement
  • Process result set
  • close connection

The following is a code example of nebula_dart_gdbc under the specification of dart_gdbc:

import 'package:nebula_dart_gdbc/nebula_dart_gdbc.dart';

void main() async {<!-- -->
  //Register driver
  DriverManager.registerDriver(NgDriver());

  // Get connection
  var conn = await DriverManager.getConnection(
    'gdbc.nebula://127.0.0.1:9669/?space=test',
    username: 'root', // username is optional
    password: 'nebula', // password is optional
  );
  
  //Create statement
  var stmt = await conn.createStatement();
  //Execute statement
  var rs = await stmt.executeQuery(gql: 'SHOW SPACES;');
  // print results
  print(rs);
  // close connection
  conn.close();
}

For now, after getting the result set, you still need to assemble the data according to the array subscript. The structure of the result set takes the form of “head array” + “data array”. In actual development, processing may be more troublesome. Currently, I am considering adding buried callbacks for object graph mapping in future versions of dart_gdbc. Or start another OGM project. Regarding this, it is currently just an idea that is not yet mature.

End

At the end of the article, I would like to share with you the journey of writing this project.

At first, I was mentally prepared for the heavy workload and was determined to give it a try. After getting started, I realized that it was not just as simple as translating the grammar, especially because the ecology of different languages is different, and the third parties referenced are also different. Even the same functions have different operating mechanisms. This part of the work The quantity will be larger than imagined.

Another is that there is no way to directly debug the database with breakpoints. It is a kind of black box call. It is like walking in the dark with weak light. Sometimes you will feel powerless when dealing with problems that you have no clue about. You can only rely on it. Make some guesses based on experience, and then verify them bit by bit. But when I connected to the actual database and got the data, I was still excited even though I had been accustomed to the process of being hit by bugs and then eliminating bugs for many years. When I started to establish the dart_gdbc specification, the “unshirkable responsibility” written in the “Preface” echoed in my mind. Until the first version was released today, this enthusiasm has been truly realized.

Finally, of course, we hope that this project can be helpful to the basic software ecosystem. I will continue to fix bugs and improve functions, hoping to become a stable and reliable project. If you are interested in the project, you are welcome to participate in the development. In addition, dart_gdbc and nebula_dart_gdbc are currently included in [Graph Community] (https://github.com/graph-cn). New open source projects about graphs will also be released here in the future. Welcome to watch.

Finally, behind this article and project are the help of two teachers, Siwei and Xiangxiao Dried Fish Steaming, thank you.

Portal

  • dart_gdbc: https://github.com/graph-cn/dart_gdbc
  • nebula_dart_gdbc: https://github.com/graph-cn/nebula_dart_gdbc
  • fbthrift: https://github.com/dudu-ltd/fbthrift

Thank you for reading this article (///▽///)

If you want to try out the graph database NebulaGraph, remember to download and use it from GitHub, (з)-☆ star it-> GitHub; exchange graph database technology and application skills with other NebulaGraph users, and leave ” Let’s play with your business card~

The 2023 NebulaGraph technology community annual essay collection event is underway. Come here to receive gifts such as Huawei Meta 60 Pro, Switch game console, Xiaomi sweeping robot, etc. Event link: https://discuss.nebula-graph.com.cn/t /topic/13970