[Go] go-es statistics interface brush count and IP access source

The number of interface brushes and IP access sources in the go-es module statistics log

  • The following is the statistics page displayed using go’s web framework gin as the backend.


Background

  • The above data comes from elk log statistics. Because elk is displayed through kibana, but kibana has a certain learning cost and cannot meet the needs of customization, so we consider using programming to process the data.
  • The first is interface statistics. Kibana’s page will only perform percentage statistics on the top500 of the field uri, and display the first 5 pieces of data. The statistics are not sufficient.
  • The second is the gateway log. The IP source collection field is through x_forward_for, which records the proxy source IP at all levels. It is not possible to directly perform data aggregation statistics on user IPs.
    • For example, here is “223.104.195.51,192.168.29.135”. For this kind of data, I need to get 223.104.195.51, because this is the user’s IP. Therefore, programming is required

Environment

  • elk 7.9
https://www.elastic.co/downloads/past-releases/elasticsearch-7-9-3
  • go 1.17, gin 1.6.3, go-elasticsearch 7.9.0
# go1.17 download address
https://go.dev/dl/

# Module download
go env -w GOPROXY=https://goproxy.cn,direct
go mod init go-ops # The go mod name of this project
go get github.com/elastic/go-elasticsearch/[email protected]
go get github.com/gin-gonic/[email protected]
  • Frontend: layui and echarts
# layui download
http://layui.dotnetcms.cn/res/static/download/layui/layui-v2.6.8.zip?v=1
# layui framework code
http://layui.dotnetcms.cn/web/demo/admin.html
# layui data table
http://layui.dotnetcms.cn/web/demo/table.html
# echarts download (requires magic)
https://cdn.jsdelivr.net/npm/[email protected]/dist/echarts.min.js
# echarts histogram
https://echarts.apache.org/handbook/zh/get-started/
#echarts pie chart
https://echarts.apache.org/handbook/en/how-to/chart-types/pie/basic-pie/
  • rear end
#gin static file service (for importing js, css, and images)
https://learnku.com/docs/gin-gonic/1.7/examples-serving-static-files/11402
#gin template engine (the front and back ends are not separated, the back-end data renders the front-end)
https://learnku.com/docs/gin-gonic/1.7/examples-html-rendering/11363
#gin binds Uri (dynamically obtains secondary routing)
https://learnku.com/docs/gin-gonic/1.7/examples-bind-uri/11391

go-elasticsearch module

  • As the name suggests, this module functions as an es client to read data from the es index. Its principle is the same as the dev tools on kibana, which is a restful api call to es.
# The following is the addition, deletion and modification documents of go-elasticsearch
https://www.elastic.co/guide/en/elasticsearch/client/go-api/current/getting-started-go.html

eql

  • The core of realizing statistical data analysis is eql (es query language). Eql is sent through the go-elasticsearch module, and then the reply body returned by es is received.
  • The following is the struct source code of the es reply body after sending eql using go-es
  • You can see that in the type Response struct, the json data we want is in the Body, but note that the type of Body is io.ReadCloser, so you need to use the io module of go to obtain the json data.
    • The code here to solve the reading problem is as follows. This function receives the es response body and returns an unserialized byte slice
// Process the es response and obtain the Body in the response body
func getResponseBody(result *esapi.Response, context *gin.Context) []byte {<!-- -->
// Receive the data returned in the es reply body. The io stream is returned here and needs to be received using the corresponding method.
var bodyBytes[]byte
bodyBytes, err := io.ReadAll(result.Body)
if err != nil {<!-- -->
panic(err)
}
return bodyBytes
}

ES client establishes connection

  • Document address
https://www.elastic.co/guide/en/elasticsearch/client/go-api/current/connecting.html
  • What is given here is the solution to https connection + username and password authentication + untrusted certificate
package esinit

import (
"github.com/elastic/go-elasticsearch/v7"
"io/ioutil"
"log"
)

var EsClient *elasticsearch.Client

func init() {<!-- -->
EsClient = newEsClient()
}

func newEsClient() *elasticsearch.Client {<!-- -->
cert, certErr := ioutil.ReadFile("esinit/es.crt") // Your untrusted https certificate
if certErr != nil {<!-- -->
log.Println(certErr)
}
EsClient, error := elasticsearch.NewClient(elasticsearch.Config{<!-- -->
Username: "your username",
Password: "your password",
Addresses: []string{<!-- -->
"https://es-cluster1:9200",
"https://es-cluster2:9200",
"https://es-cluster3:9200",
},
CACert: cert,
})
if error != nil {<!-- -->
panic(error)
}
return EsClient
}

Three sections of eql to implement statistics

  • Here eql uses the fmt.Sprintf() method for parameter passing
  • The first paragraph of eql is statistical PV
    • Note here that we are in the East Eighth District, so the statistical pv starts at 16:00. Here, “aggs” aggregation statistics are performed based on the @timestamp field.
func getPvResponse(startYear int, startMonth int, startDay int, endYear, endMonth int, endDay int) *esapi.Response {<!-- -->
query := fmt.Sprintf(`
{
  "query": {
"bool": {
"must": [
{
"range": {
"@timestamp": {
"gte": "%d- d- dT16:00:00",
"lte": "%d- d- dT16:00:00"
}
}
}
]
}
  },
  "aggs": {
"log_count": {
"value_count": {
"field": "@timestamp"
}
}
  }
}
`, startYear, startMonth, startDay, endYear, endMonth, endDay)
result, _ := esinit.EsClient.Search(
esinit.EsClient.Search.WithIndex("k8s-istio-ingress*"), // Index name
esinit.EsClient.Search.WithBody(strings.NewReader(query)), // eql
)
return result
}
  • The second paragraph is the aggregate statistics of the interface (uri field) of the microservice (java)
    • The sortUri used here is the binding uri function of gin (dynamically obtains the name of the secondary route)
    • Here returns the uri field data of 10,000 es documents in the previous day
func getSortResponse(context *gin.Context) *esapi.Response {<!-- -->
if err := context.ShouldBindUri( & amp;sortUri); err != nil {<!-- --> // sortUri secondary route, passing the index name
context.JSON(400, gin.H{<!-- -->"msg": err})
}
//Search documents
// eql searches 10,000 records within the time range and only displays the contents of the uri field
query := fmt.Sprintf(`
{
"_source": ["uri"],
"query": {
"bool": {
"filter": [
{
"range": {
"@timestamp": {
"gte": "now-1d/d",
"lte": "now/d"
}
}
}
]
}
},
"size": 10000
}
`)
// Search corresponding index
result, _ := esinit.EsClient.Search(
esinit.EsClient.Search.WithIndex(sortUri.Name + "*"),
esinit.EsClient.Search.WithBody(strings.NewReader(query)),
)
return result
}
  • The third paragraph is the IP request statistics of istio in the previous hour, returning 2000 records
func getIstioDataResponse() *esapi.Response {<!-- -->
query := `
{
  "_source": [
    "x_forwarded_for",
    "@timestamp",
    "path",
    "user_agent_a",
    "response_code",
    "method",
    "upstream_cluster"
  ],
  "query": {
    "bool": {
      "filter": [
        {
          "range": {
            "@timestamp": {
              "gte": "now-1h/h",
              "lte": "now/d"
            }
          }
        }
      ]
    }
  },
  "size": 2000
}
`
result, _ := esinit.EsClient.Search(
esinit.EsClient.Search.WithIndex("k8s-istio-ingress*"),
esinit.EsClient.Search.WithBody(strings.NewReader(query)),
)
return result
}
  • The above eql function will return a []byte slice, which can be processed by json.Unmarshal or other struct-to-json modules to obtain the data. Then you can render