1. DSL category query
Elasticsearch provides a JSON-based DSL (Domain Specific Language) to define queries. Common query types include:
-
Query all: query all data, for general testing. For example: match_all
-
Full text search: use the word segmenter to segment the user input content, and then match it in the inverted index database. For example:
- match_query
- multi_match_query
-
Precise query: Search for data based on precise entry values, generally for keyword, numeric, date, boolean and other types of fields. For example:
- ids
- range
- term
-
Geographic (geo) query: query based on latitude and longitude. For example:
- geo_distance
- geo_bounding_box
-
Compound query: compound query can combine the above-mentioned various query conditions and combine query conditions. For example:
- bool
- function_score
2. Deep paging problem
For deep paging, ES provides two solutions, official documents:
- search after: sorting is required when paging, the principle is to query the next page of data starting from the last sorting value. The official recommended way to use.
- scroll: The principle is to form a snapshot of the sorted document ids and store them in memory. It is officially deprecated.
Common implementation schemes and advantages and disadvantages of pagination query:
-
from + size:- Advantages: Support random page turning
- Disadvantages: deep paging problem, the default query upper limit (from + size) is 10000
- Scenario: Random page-turning searches such as Baidu, JD.com, Google, and Taobao
-
search after:- Advantages: no query upper limit (the size of a single query does not exceed 10000)
- Disadvantage: can only query backward page by page, does not support random page turning
- Scenario: Search without random page turning requirements, such as mobile phone scrolling down to turn pages
-
scroll:- Advantages: no query upper limit (the size of a single query does not exceed 10000)
- Disadvantages: There will be additional memory consumption, and the search results are not real-time
- Scenario: Acquisition and migration of massive data. It is not recommended starting from ES7.1. It is recommended to use the search after solution.
3. DSL classification query code implementation
- Array traversal uses stream
List<HotelDoc> hotelDocList = Stream.of(hits).map(hit -> {<!-- -->
//Get the json string corresponding to each record
String json = hit. getSourceAsString();
HotelDoc hotelDoc = JSON. parseObject(json, HotelDoc. class);
System.out.println("hotelDoc = " + hotelDoc);
return hotelDoc;
}).collect(Collectors.toList());
- DSL classification query code implementation
@Autowired
private RestHighLevelClient client;
/**
* Query all
* match_all
*
* @throws Exception
*/
@Test
public void matchAllTest() throws Exception {
//1. Create a request object
SearchRequest searchRequest = new SearchRequest(ESConstant.HOTEL_INDEX);
//2.. Set conditions, prepare DSL
searchRequest.source().query(
QueryBuilders.matchAllQuery()
);
//3. Execute query, search
SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
//4. Parse the result
//4.1 The total number of records obtained
Long count = searchResponse.getHits().getTotalHits().value;
System.out.println("count = " + count);
//4.2 Get the hits array (record list)
SearchHit[] hits = searchResponse. getHits(). getHits();
List<HotelDoc> hotelDocList = Stream.of(hits).map(hit -> {<!-- -->
//Get the json string corresponding to each record
String json = hit. getSourceAsString();
HotelDoc hotelDoc = JSON. parseObject(json, HotelDoc. class);
System.out.println("hotelDoc = " + hotelDoc);
return hotelDoc;
}).collect(Collectors.toList());
/*for (SearchHit hit : hits) {
//Get the json string corresponding to each record
String json = hit. getSourceAsString();
HotelDoc hotelDoc = JSON. parseObject(json, HotelDoc. class);
System.out.println("hotelDoc = " + hotelDoc);
}*/
}
/**
* match
* multi_match
*
* @throws Exception
*/
@Test
public void matchTest() throws Exception {
//1. Create a request object
SearchRequest searchRequest = new SearchRequest(ESConstant.HOTEL_INDEX);
//2. Set request conditions, DSL
searchRequest.source().query(
//QueryBuilders.matchQuery("all", "Home on the Bund")
QueryBuilders.multiMatchQuery("Home on the Bund", "name", "brand", "business")
);
//3. Execute query
SearchResponse response = client.search(searchRequest, RequestOptions.DEFAULT);
//4. Parse the result
Long count = response.getHits().getTotalHits().value;
System.out.println("count = " + count);
SearchHit[] hits = response. getHits(). getHits();
for (SearchHit hit : hits) {
String json = hit. getSourceAsString();
System.out.println("json = " + json);
}
}
/**
* term precise query
*
* @throws Exception
*/
@Test
public void termTest() throws Exception {
//1. Create a request object
SearchRequest searchRequest = new SearchRequest(ESConstant.HOTEL_INDEX);
//2. Set request conditions, DSL
searchRequest. source()
.query(QueryBuilders.termQuery("city", "Shanghai"));
//3. Execute query
SearchResponse response = client.search(searchRequest, RequestOptions.DEFAULT);
//4. Parse the result
Long count = response.getHits().getTotalHits().value;
System.out.println("count = " + count);
SearchHit[] hits = response. getHits(). getHits();
for (SearchHit hit : hits) {
String json = hit. getSourceAsString();
System.out.println("json = " + json);
}
}
/**
* range
* range query
*
* @throws Exception
*/
@Test
public void rangeTest() throws Exception {
//1. Create a request object
SearchRequest searchRequest = new SearchRequest(ESConstant.HOTEL_INDEX);
//2. Set request conditions, DSL
searchRequest.source().query(QueryBuilders.rangeQuery("price").gte(1000).lt(3000));
//3. Execute query
SearchResponse response = client.search(searchRequest, RequestOptions.DEFAULT);
//4. Parse the result
Long count = response.getHits().getTotalHits().value;
System.out.println("count = " + count);
SearchHit[] hits = response. getHits(). getHits();
for (SearchHit hit : hits) {
String json = hit. getSourceAsString();
System.out.println("json = " + json);
}
}
/**
* Longitude and latitude query
*
* @throws Exception
*/
@Test
public void geoDistanceTest() throws Exception {
//1. Create a request object
SearchRequest searchRequest = new SearchRequest(ESConstant.HOTEL_INDEX);
//2. Set request conditions, DSL
searchRequest. source()
.query(QueryBuilders. geoDistanceQuery("location")
//.point(new GeoPoint(31.242201,121.509106))
.point(31.242201, 121.509106)
.distance(15, DistanceUnit.KILOMETERS));
//3. Execute query
SearchResponse response = client.search(searchRequest, RequestOptions.DEFAULT);
//4. Parse the result
Long count = response.getHits().getTotalHits().value;
System.out.println("count = " + count);
SearchHit[] hits = response. getHits(). getHits();
for (SearchHit hit : hits) {
String json = hit. getSourceAsString();
System.out.println("json = " + json);
}
}
/**
* bool query
*
* @throws Exception
*/
@Test
public void Test() throws Exception {
//1. Create a request object
SearchRequest searchRequest = new SearchRequest(ESConstant.HOTEL_INDEX);
//2. Set request conditions, DSL
searchRequest. source()
.query(QueryBuilders. boolQuery()
.must(QueryBuilders.matchQuery("name", "Home Inn"))
.mustNot(QueryBuilders. rangeQuery("price").gt(400))
.filter(QueryBuilders. geoDistanceQuery("location")
.point(31.21, 121.5)
.distance(15, DistanceUnit.KILOMETERS)));
//3. Execute query
SearchResponse response = client.search(searchRequest, RequestOptions.DEFAULT);
//4. Parse the result
Long count = response.getHits().getTotalHits().value;
System.out.println("count = " + count);
SearchHit[] hits = response. getHits(). getHits();
for (SearchHit hit : hits) {
String json = hit. getSourceAsString();
System.out.println("json = " + json);
}
}
Filter + score + sort + distance sort + highlight + pagination
The obtained distance data and highlight data need special processing
@Autowired
private RestHighLevelClient client;
@Override
public PageResult<HotelDoc> searchHotel(SearchParam searchParam) {<!-- -->
//1. Verify parameters
//The first few data
Integer page = searchParam. getPage();
//page size
Integer size = searchParam. getSize();
//Search content
String key = searchParam. getKey();
//Create a wrapper result object
PageResult<HotelDoc> hotelDocPageResultge = new PageResult<>();
//Create collection
List<HotelDoc> hotelDocList = new ArrayList<>();
try {<!-- -->
if (page == null || size == null) {<!-- -->
throw new RuntimeException("Illegal parameter");
}
//2. Business processing
//2.1 Create request object
SearchRequest searchRequest = new SearchRequest(ESConstant.HOTEL_INDEX);
//2.2 Prepare query conditions
//Create query condition object---->Use bool query
BoolQueryBuilder boolQuery = QueryBuilders. boolQuery();
//------> search box judges null and empty string
if (StringUtils.isBlank(key)) {<!-- -->
//2.2.1 If it is empty, query all
boolQuery. must(QueryBuilders. matchAllQuery());
} else {<!-- -->
//2.2.2 If it is not empty, query according to the condition
boolQuery. must(QueryBuilders. matchQuery("all", key));
}
//------> city
String city = searchParam. getCity();
if (StringUtils.isNoneBlank(city)) {<!-- -->
boolQuery. filter(QueryBuilders. termQuery("city", city));
}
//------------>Brand
String brand = searchParam. getBrand();
if (StringUtils.isNoneBlank(brand)) {<!-- -->
boolQuery. filter(QueryBuilders. termQuery("brand", brand));
}
//------------> star rating
String starName = searchParam. getStarName();
if (StringUtils.isNoneBlank(starName)) {<!-- -->
boolQuery. filter(QueryBuilders. termQuery("starName", starName));
}
//---------> price
Integer minPrice = searchParam. getMinPrice();
Integer maxPrice = searchParam. getMaxPrice();
if (null != minPrice & amp; & amp; null != maxPrice) {<!-- -->
boolQuery. filter(QueryBuilders
.rangeQuery("price")
.gte(minPrice)
.lte(maxPrice));
}
//-----------> score isAD=true add points for advertising
FunctionScoreQueryBuilder functionScoreQueryBuilder = new FunctionScoreQueryBuilder(boolQuery,
new FunctionScoreQueryBuilder.FilterFunctionBuilder[]{<!-- -->
new FunctionScoreQueryBuilder. FilterFunctionBuilder(
QueryBuilders. termQuery("isAD", true),
ScoreFunctionBuilders. weightFactorFunction(100F)
)}).boostMode(CombineFunction.SUM);
//searchRequest.source().query(boolQuery);
//=================> Sort before pagination, after condition
String sortBy = searchParam. getSortBy();
if ("score".equals(sortBy)) {<!-- -->
searchRequest.source().sort("score", SortOrder.DESC);
} else if ("price".equals(sortBy)) {<!-- -->
searchRequest.source().sort("price", SortOrder.DESC);
}
//================> Sort according to distance --> Need to process after getting the query distance
String location = searchParam. getLocation();
if (StringUtils.isNoneBlank(location)) {<!-- -->
searchRequest.source().sort(
SortBuilders. geoDistanceSort("location", new GeoPoint(location))
.unit(DistanceUnit.KILOMETERS));
}
//2.3 Prepare paging parameters
//============================>highlight --> highlight data also needs to be processed and returned to the front end
//functionScoreQueryBuilder encapsulates boolQuery
searchRequest. source()
.query(functionScoreQueryBuilder).highlighter(
new HighlightBuilder()
.field("name")
.requireFieldMatch(false)
.preTags("<font style='color:red'>")
.postTags("</font>")
)
.from((page - 1) * size)
.size(size);
//2.4 execute query
SearchResponse response = client.search(searchRequest, RequestOptions.DEFAULT);
//2.5 parse the result and encapsulate the data
//2.5.1 Get the total number of records that meet the conditions
long total = response.getHits().getTotalHits().value;
//2.5.2 Get the number of records that meet the conditions
for (SearchHit hit : response. getHits(). getHits()) {<!-- -->
String hotelDocJson = hit. getSourceAsString();
HotelDoc hotelDoc = JSON.parseObject(hotelDocJson, HotelDoc.class);
//Process the distance data obtained
Object[] sortValues = hit. getSortValues();
if (sortValues != null & amp; & amp; sortValues. length > 0) {<!-- -->
//Get the distance between the current position and location
Object distance = sortValues[0];
hotelDoc. setDistance(distance);
}
// process the highlighted data
Map<String, HighlightField> highlightFields = hit. getHighlightFields();
//Judge whether the highlighted data obtained is empty
if (highlightFields!=null & amp; & amp; highlightFields. size()>0) {<!-- -->
HighlightField highlightField = highlightFields. get("name");
Text[] fragments = highlightField. getFragments();
String name = fragments[0].toString();
hotelDoc. setName(name);
}
hotelDocList.add(hotelDoc);
}
//2.5.3 Encapsulation data
hotelDocPageResultge.setTotal(total);
hotelDocPageResultge.setHotels(hotelDocList);
} catch (IOException e) {<!-- -->
e.printStackTrace();
}
return hotelDocPageResult;
}