This article brings you relevant knowledge about java, which mainly introduces issues related to the open source library SPL for structured data processing. Let’s take a look at the ideal structure under java. Data processing class library, I hope it will be helpful to everyone.

Recommended study: "java Video Tutorial"

Modern Java application architecture increasingly emphasizes the separation of data storage and processing. Obtain better maintainability, scalability and portability, such as the popular microservices. This architecture usually requires business logic to be implemented in Java programs instead of being placed in the database as in traditional application architecture.

Most of the business logic in applications involves structured data processing. The database (SQL) has rich support for such tasks, and business logic can be implemented relatively easily. However, Java has always lacked such basic support, making it very cumbersome and inefficient to implement business logic in Java. As a result, despite various architectural advantages, development efficiency has dropped significantly.

If we also provide a complete set of structured data processing and calculation class libraries in Java, then this problem can be solved: enjoying the advantages of the architecture without reducing development efficiency.

What kind of abilities are needed?

What characteristics should an ideal structured data processing class library under Java have? We can summarize it from SQL:

1 Set computing capability

Structured data often appears in batches (in the form of sets). In order to conveniently calculate this type of data , it is necessary to provide sufficient set computing capabilities.

If there is no set operation library, there is only the basic data type of array (equivalent to a set). If we want to do a simple sum of the set members, we also need to write four or five lines of loop statements to complete, filter, and group. Operations such as aggregation require hundreds of lines of code.

SQL provides a rich set of operations, such as SUM/COUNT and other aggregation operations, WHERE is used for filtering, GROUP is used for grouping, and it also supports basic operations such as intersection, union, and difference for sets. The code written in this way will be much shorter.

2 Lambda syntax

Is it enough to have set operation capabilities? If we develop a batch of set operation libraries for Java, can we achieve the effect of SQL?

It’s not that simple!

Take filtering operation as an example. Filtering usually requires a condition to retain the set members that meet the condition. In SQL, this condition appears in the form of an expression. For example, writing WHERE x>0 means retaining those members that make the calculation result of x>0 true. The expression x>0 is not evaluated before executing this statement, but is evaluated for each set member during the iteration. Essentially, this expression is essentially a function, a function that takes the current collection members as parameters. For the WHERE operation, it is equivalent to using a function defined with an expression as a parameter of WHERE.

There is a term for this way of writing called Lambda grammar, or functional language.

Without Lambda syntax, we would often have to temporarily define functions, which would make the code very cumbersome and prone to name conflicts.

Lambda syntax is widely used in SQL. It is not required for filtering and grouping operations. It can also be used in unnecessary scenarios such as calculated columns, which greatly simplifies the code.

3 Directly refer to fields in Lambda syntax

Structured data is not a simple single value, but a record with fields.

We found that when referencing record fields in SQL expression parameters, in most cases you can use the field name directly without specifying the record to which the field belongs. Only when there are multiple fields with the same name, you need to use the table name ( or alias) to distinguish.

Although the new version of Java has also begun to support Lambda syntax, it can only pass the current record as a parameter into the function defined with Lambda syntax, and then always bring this record when writing calculation formulas. For example, when calculating the amount using unit price and quantity, if the parameter used to represent the current member is named x, it needs to be written in the long-winded form of "x. unit price * x. quantity". In SQL, it can be written more intuitively as "unit price * quantity".

4 Dynamic data structure

SQL can also support dynamic data structures very well.

In structured data calculations, the return value is often structured data, and the result data structure is related to operations and cannot be prepared before writing the code. Therefore, it is necessary to support dynamic data structure capabilities.

Any SELECT statement in SQL will generate a new data structure. You can add and delete fields at will in the code without defining the structure (class) in advance. This is not possible with languages ??like Java. All structures (classes) used must be defined during the code compilation phase. In principle, new structures cannot be dynamically generated during execution.

5 Interpreted Language

From the analysis of the previous articles, we can already draw the conclusion: Java itself is not suitable as a language for structured data processing. Its Lambda mechanism does not support feature 3, and as a compiled language, it cannot implement feature 4.

In fact, the Lambda syntax mentioned earlier is not suitable for implementation in compiled languages. The compiler cannot determine whether the expression written to the parameter position should calculate the value of the expression on the spot and then pass it on, or whether it should compile the entire expression into a function and pass it, and more syntax symbols need to be designed to distinguish it. Interpreted languages ??do not have this problem. Whether the expression as a parameter is calculated first or is calculated after traversing the set members can be decided by the function itself.

SQL is indeed an interpreted language.

Introducing SPL

Stream is a structured data processing library launched in an official capacity in Java 8, but it does not meet the above requirements. It does not have professional structured data types, lacks many important structured data calculation functions, is not an interpreted language, does not support dynamic data types, and the interface of Lambda syntax is complex.

Kotlin is part of the Java ecosystem. It has made slight improvements based on Stream and also provides structured data calculation types. However, due to insufficient structured data calculation functions, it is not An interpreted language does not support dynamic data types, and the interface of Lambda syntax is complex. It is still not an ideal structured data computing class library.

Scala provides a rich set of structured data calculation functions, but the characteristics of compiled languages ??also prevent it from becoming an ideal structured data calculation class library.

So, what else can be used in the Java ecosystem?

esProc SPL.

SPL is a programming language interpreted and executed by Java. It has a rich structured data calculation class library, simple Lambda syntax and convenient and easy-to-use dynamic data structures. It is an ideal structured processing class library under Java.

Rich set operation functions

SPL provides a professional structured data type, namely sequence table. Like the SQL data table, the sequence table is a collection of batch records and has the general functions of structured data types. The following is an example.

Parse the source data and generate the sequence table:

Orders=T("d:/Orders.csv")

Generate a new sequence table from the original sequence table by column name:

Orders.new(OrderID,?Amount,?OrderDate)

Calculated column:

Orders.new(OrderID,?Amount,?year(OrderDate))

Field rename:

Orders.new(OrderID:ID,?SellerId,?year(OrderDate):y)

Use fields by serial number:

Orders.groups(year(_5),_2;?sum(_4))

Sequence table rename (left association)

join@1(Orders:o,SellerId?;?Employees:e,EId).groups(e.Dept;?sum(o.Amount))

Sequence table supports all structured calculation functions, and the calculation results are also It is also a sequence table, not a data type such as Map. For example, continue to process structured data for grouped summary results:

Orders.groups(year(OrderDate):y;?sum(Amount):m).new(y:OrderYear,?m*0.2:discount)

Based on the sequence table, SPL provides a wealth of structured data calculation functions, such as filtering, sorting, grouping, deduplication, and renaming. , calculated columns, associations, subqueries, set calculations, ordered calculations, etc. These functions have powerful computing capabilities and can complete calculations independently without hard-coding assistance:

Combined query:

Orders.select(Amount>1000?&&?Amount<=3000 && like(Client,"*bro*"))

Sort:

Orders.sort(-Client,Amount)

Group summary:

Orders.groups(year(OrderDate),Client; sum(Amount))

Internal association:

join(Orders:o,SellerId ; Employees:e,EId).groups(e.Dept; sum(o.Amount))

Simple Lambda syntax

SPL supports simple Lambda syntax. There is no need to define function names and function bodies. Expressions can be used directly. As parameters of functions, such as filtering:

Orders.select(Amount>1000)

When modifying business logic, there is no need to reconstruct the function, just simply modify the expression:

Orders.select(Amount>1000?&&?Amount<2000)

SPL is an interpreted language and uses parameter expressions It is not necessary to explicitly define parameter types, making the Lambda interface simpler. For example, to calculate the sum of squares, if you want to calculate the square during the sum process, you can write it intuitively:

Orders.sum(Amount*Amount)

Similar to SQL, SPL syntax also supports the direct use of field names in single-table calculations:

Orders.sort(-Client, Amount)

Dynamic Data Structure

SPL is an interpreted language that naturally supports dynamic data structures and can dynamically generate new sequences based on the calculation result structure. It is particularly suitable for calculations such as calculated columns, group summarization, and correlation. For example, directly recalculating the results of group summary:

Orders.groups(Client;sum(Amount):amt).select(amt>1000?&&?like(Client,"*S*"))

or directly recalculating the results of correlation calculation:

join(Orders:o,SellerId?;?Employees:e,Eid).groups(e.Dept;?sum(o.Amount))

More complex ones Calculations are usually divided into multiple steps, and the data structure of each intermediate result is almost different. SPL supports dynamic data structures without having to define the structure of these intermediate results first. For example, based on the customer payment record table of a certain year, calculate the top 10 customers with monthly payment amount:

Sales2021.group(month(sellDate)).(~.groups(Client;sum(Amount):sumValue)).(~.sort(-sumValue))?.(~.select(#<=10)).(~.(Client)).isect()

Directly execute SQL

SPL It also implements a SQL interpreter, which can directly execute SQL. It can support everything from basic WHERE, GROUP to JOIN, and even WITH:

$select * from d:/Orders.csv where (OrderDate<date(&#39;2020-01-01&#39;) and Amount<=100)or (OrderDate>=date('2020-12-31')?and?Amount>100)

$select?year(OrderDate),Client?,sum(Amount),count(1)?from?d:/Orders.csv
group?by?year(OrderDate),Client
having?sum(Amount)<=100

$select o.OrderId,o.Client,e.Name e.Dept from d:/Orders.csv o
join d:/Employees.csv e on o.SellerId=e.Eid

$with t as (select Client ,sum(amount) s from d:/Orders.csv group by Client)
select t.Client, t.s, ct.Name, ct.address from t
left join ClientTable ct on t.Client=ct.Client

More language advantages

As a professional structure As a data processing language, SPL not only covers all the computing capabilities of SQL, but also has more powerful advantages in terms of language:

discreteness and its support for more thorough aggregation

Collection is a basic feature of SQL, which supports data to participate in operations in the form of sets. However, the discrete nature of SQL is very poor. All set members must participate in the operation as a whole and cannot be separated from the set. High-level languages ??such as Java support good discreteness, and array members can be operated independently.

但是，更徹底的集合化需要離散性來(lái)支持，集合成員可以游離在集合之外，并與其它數(shù)據(jù)隨意構(gòu)成新的集合參與運(yùn)算 。

SPL兼具了SQL的集合化和Java的離散性，從而可以實(shí)現(xiàn)更徹底的集合化。

比如，SPL中很容易表達(dá)“集合的集合”，適合分組后計(jì)算。比如，找到各科成績(jī)均在前10名的學(xué)生：

Employees.select(性別:"男",部門.經(jīng)理.性別:"女")

有序計(jì)算是離散性和集合化的典型結(jié)合產(chǎn)物，成員的次序在集合中才有意義，這要求集合化，有序計(jì)算時(shí)又要將每個(gè)成員與相鄰成員區(qū)分開，會(huì)強(qiáng)調(diào)離散性。SPL兼具集合化和離散性，天然支持有序計(jì)算。

具體來(lái)說(shuō)，SPL可以按絕對(duì)位置引用成員，比如，取第3條訂單可以寫成Orders(3)，取第1、3、5條記錄可以寫成Orders([1,3,5])。

SPL也可以按相對(duì)位置引用成員，比如，計(jì)算每條記錄相對(duì)于上一條記錄的金額增長(zhǎng)率：Orders.derive(amount/amount[-1]-1)

SPL還可以用#代表當(dāng)前記錄的序號(hào)，比如把員工按序號(hào)分成兩組，奇數(shù)序號(hào)一組，偶數(shù)序號(hào)一組：Employees.group(#%2==1)

更方便的函數(shù)語(yǔ)法

大量功能強(qiáng)大的結(jié)構(gòu)化數(shù)據(jù)計(jì)算函數(shù)，這本來(lái)是一件好事，但這會(huì)讓相似功能的函數(shù)不容易區(qū)分。無(wú)形中提高了學(xué)習(xí)難度。

SPL提供了特有的函數(shù)選項(xiàng)語(yǔ)法，功能相似的函數(shù)可以共用一個(gè)函數(shù)名，只用函數(shù)選項(xiàng)區(qū)分差別。比如select函數(shù)的基本功能是過(guò)濾，如果只過(guò)濾出符合條件的第1條記錄，只須使用選項(xiàng)@1：

Orders.select@1(Amount>1000)

數(shù)據(jù)量較大時(shí)，用并行計(jì)算提高性能，只須改為選項(xiàng)@m：

Orders.select@m(Amount>1000)

對(duì)排序過(guò)的數(shù)據(jù)，用二分法進(jìn)行快速過(guò)濾，可用@b：

Orders.select@b(Amount>1000)

函數(shù)選項(xiàng)還可以組合搭配，比如：

Orders.select@1b(Amount>1000)

結(jié)構(gòu)化運(yùn)算函數(shù)的參數(shù)常常很復(fù)雜，比如SQL就需要用各種關(guān)鍵字把一條語(yǔ)句的參數(shù)分隔成多個(gè)組，但這會(huì)動(dòng)用很多關(guān)鍵字，也使語(yǔ)句結(jié)構(gòu)不統(tǒng)一。

SPL支持層次參數(shù)，通過(guò)分號(hào)、逗號(hào)、冒號(hào)自高而低將參數(shù)分為三層，用通用的方式簡(jiǎn)化復(fù)雜參數(shù)的表達(dá)：

join(Orders:o,SellerId?;?Employees:e,EId)

擴(kuò)展的Lambda語(yǔ)法

普通的Lambda語(yǔ)法不僅要指明表達(dá)式（即函數(shù)形式的參數(shù)），還必須完整地定義表達(dá)式本身的參數(shù)，否則在數(shù)學(xué)形式上不夠嚴(yán)密，這就讓Lambda語(yǔ)法很繁瑣。比如用循環(huán)函數(shù)select過(guò)濾集合A，只保留值為偶數(shù)的成員，一般形式是：

A.select(f(x):{x%2==0}?)

這里的表達(dá)式是x%2==0，表達(dá)式的參數(shù)是f(x)里的x，x代表集合A里的成員，即循環(huán)變量。

SPL用固定符號(hào)~代表循環(huán)變量，當(dāng)參數(shù)是循環(huán)變量時(shí)就無(wú)須再定義參數(shù)了。在SPL中，上面的Lambda語(yǔ)法可以簡(jiǎn)寫作：A.select(~ %2==0)

普通Lambda語(yǔ)法必須定義表達(dá)式用到的每一個(gè)參數(shù)，除了循環(huán)變量外，常用的參數(shù)還有循環(huán)計(jì)數(shù)，如果把循環(huán)計(jì)數(shù)也定義到Lambda中，代碼就更繁瑣了。

SPL用固定符號(hào)#代表循環(huán)計(jì)數(shù)變量。比如，用函數(shù)select過(guò)濾集合A，只保留序號(hào)是偶數(shù)的成員，SPL可以寫作：A.select(# %2==0)

相對(duì)位置經(jīng)常出現(xiàn)在難度較大的計(jì)算中，而且相對(duì)位置本身就很難計(jì)算，當(dāng)要使用相對(duì)位置時(shí)，參數(shù)的寫法將非常繁瑣。

SPL用固定形式[序號(hào)]代表相對(duì)位置：

無(wú)縫集成、低耦合、熱切換

作為用Java解釋的腳本語(yǔ)言，SPL提供了JDBC驅(qū)動(dòng)，可以無(wú)縫集成進(jìn)Java應(yīng)用程中。

簡(jiǎn)單語(yǔ)句可以像SQL一樣直接執(zhí)行：

…
Class.forName("com.esproc.jdbc.InternalDriver");
Connection?conn?=DriverManager.getConnection("jdbc:esproc:local://");
PrepareStatement?st?=?conn.prepareStatement("=T(\"D:/Orders.txt\").select(Amount>1000?&&?Amount<=3000?&&?like(Client,\"*S*\"))");
ResultSet?result=st.execute();
...

復(fù)雜計(jì)算可以存成腳本文件，以存儲(chǔ)過(guò)程方式調(diào)用

…
Class.forName("com.esproc.jdbc.InternalDriver");
Connection?conn?=DriverManager.getConnection("jdbc:esproc:local://");
Statement?st?=?connection.();
CallableStatement?st?=?conn.prepareCall("{call?splscript1(?,??)}");
st.setObject(1,?3000);
st.setObject(2,?5000);?
ResultSet?result=st.execute();
...

將腳本外置于Java程序，一方面可以降低代碼耦合性，另一方面利用解釋執(zhí)行的特點(diǎn)還可以支持熱切換，業(yè)務(wù)邏輯變動(dòng)時(shí)只要修改腳本即可立即生效，不像使用Java時(shí)常常要重啟整個(gè)應(yīng)用。這種機(jī)制特別適合編寫微服務(wù)架構(gòu)中的業(yè)務(wù)處理邏輯。

推薦學(xué)習(xí)：《java視頻教程》

The above is the detailed content of Let you understand the Java structured data processing open source library SPL. For more information, please follow other related articles on the PHP Chinese website!