sentinel and sentinel-golang make your service rock solid

First of all, let’s talk about Sentinel. This is a flow control software that has been used internally by Alibaba for many years and evolved. It has withstood the test of Double Eleven for many years. It was first used for the Java language and the Sentinel-golang version was launched in 2020.

Official documentation: https://sentinelguard.io/zh-cn/docs/introduction.html

Sentinel basic concepts

Resources

Resources are a key concept in Sentinel. It can be anything within a Java application, such as a service provided by the application, a service provided by another application called by the application, or even a piece of code. In the following documentation, we will use resources to describe code blocks.

As long as the code defined through the Sentinel API is a resource, it can be protected by Sentinel. In most cases, you can use method signatures, URLs, or even service names as resource names to identify resources.

Rules

Rules set around the real-time status of resources can include flow control rules, circuit breaker degradation rules, and system protection rules. All rules can be dynamically adjusted in real time.

Sentinel functions and design concepts

Flow control

Flow control is a commonly used concept in network transmission, which is used to adjust the sending data of network packets. However, from the perspective of system stability, there are also many considerations in the speed of processing requests. Requests arriving at any time are often random and uncontrollable, and the system’s processing capacity is limited. We need to control the flow based on the system’s processing capabilities. Sentinel, as a dispatcher, can adjust random requests into appropriate shapes as needed, as shown in the following figure:

Flow control has the following perspectives:

  • The calling relationship of resources, such as the calling link of resources and the relationship between resources;
  • Operational indicators, such as QPS, thread pool, system load, etc.;
  • Control effects, such as direct current limiting, cold start, queuing, etc.

The design concept of Sentinel allows you to freely choose the angle of control and flexibly combine it to achieve the desired effect.

Circuit breaker downgrade

What is circuit breaker downgrade

In addition to flow control, reducing unstable resources in the call link is also one of Sentinel’s missions. Due to the complexity of the calling relationship, if a resource in the calling link becomes unstable, requests will eventually accumulate. This problem is the same as the one described in Hystrix.

The principles of Sentinel and Hystrix are the same: when a resource in the calling link becomes unstable, for example, manifested as timeout and the exception ratio increases, the calling of this resource will be restricted and the request will fail quickly to avoid It affects other resources and ultimately produces an avalanche effect.

Fusion degradation design concept

Sentinel and Hystrix take completely different approaches when it comes to restrictions.

Hystrix uses thread pools to isolate dependencies (corresponding resources in our concept). The advantage of this is that the most complete isolation between resources is achieved. The disadvantage is that in addition to increasing the cost of thread switching, it is also necessary to allocate the thread pool size to each resource in advance.

Sentinel takes two approaches to this problem:

  • Limit by number of concurrent threads

Different from the resource pool isolation method, Sentinel reduces the impact of unstable resources on other resources by limiting the number of concurrent resource threads. This not only eliminates the cost of thread switching, but also does not require you to pre-allocate the size of the thread pool. When a resource becomes unstable, for example, the response time becomes longer, the direct impact on the resource is the gradual accumulation of threads. When the number of threads accumulates on a specific resource to a certain number, new requests for that resource will be rejected. The stacked threads complete their tasks before continuing to receive requests.

  • Degrade resources by response time

In addition to controlling the number of concurrent threads, Sentinel can quickly degrade unstable resources through response time. When the response time of a dependent resource is too long, all access to the resource will be directly denied and will not be restored until the specified time window has passed.

System load protection

Sentinel also provides system-dimensional adaptive protection capabilities. Preventing avalanches is an important part of system protection. When the system load is high, if requests continue to come in, the system may crash and become unable to respond. In a cluster environment, network load balancing will forward the traffic that should be carried by this machine to other machines. If other machines are also in an edge state at this time, the increased traffic will cause this machine to crash, and eventually the entire cluster will become unavailable.

In response to this situation, Sentinel provides a corresponding protection mechanism to achieve a balance between the system’s inlet traffic and the system’s load, ensuring that the system can handle the most requests within its capabilities.

How Sentinel works

The main working mechanism of Sentinel is as follows:

  • Provides adaptation or display APIs for mainstream frameworks to define resources that need to be protected, and provides facilities for real-time statistics of resources and call link analysis.
  • Control traffic according to preset rules and real-time statistical information on resources. At the same time, Sentinel provides an open interface to facilitate you to define and change rules.
  • Sentinel provides a real-time monitoring system to help you quickly understand the current system status.

Flow control degradation and fault tolerance standards

The Sentinel community is extracting traffic management related standards into OpenSergo spec, and Sentinel is implemented as a traffic management standard. For the latest progress of Sentinel flow control degradation and fault tolerance spec, please refer to opensergo-specification. The community is also welcome to work together to improve the standard and implementation.

Sentinel Go Practice

Introduce dependencies

> go get -u github.com/alibaba/sentinel-golang

Adapted framework

  • Web
    • CloudWeGo Hertz
    • echo
    • gear
    • Gin
  • RPC
    • CloudWeGo Kitex
    • dubbo-go
    • gRPC-go
    • go-micro
    • go-zero
  • Service Mesh & amp; & amp; Runtime
    • Dapr
    • MOSN

Define resources

Resource is one of the core concepts in Sentinel. All current limiting and circuit breaker mechanisms in Sentinel are effective based on resources. The current limiting and circuit breaker rules of different resources are isolated from each other and do not affect each other.

In Sentinel, users can flexibly define resource burying points. A resource can be an application, interface, function, or even a piece of code. Our traffic management mechanism is to protect this resource from running as expected.

Users can package resource access through the interface in the Sentinel api package. This step is called “buried point”. Each buried point has a resource name (resource), which means that the call or access to this resource was triggered. After we have resource burying points, we can configure traffic management rules for resource burying points. Even if no rules are configured, resource burying will still generate metric statistics.

gin framework

> go get -u github.com/sentinel-group/sentinel-go-adapters/gin
go: github.com/sentinel-group/[email protected] requires
        github.com/micro/go-micro/[email protected] requires
        github.com/micro/cli/[email protected]: reading https://goproxy.io/github.com/micro/cli/v2/@v/v2.1.2.mod: 404 Not Found
        server response:
        not found: github.com/micro/cli/[email protected]: invalid version: git ls-remote -q origin in /data1/golang/pkg/mod/cache/vcs/2f5431eb5439e9d79f82a6d853348656f17b78125db9eda81300
bc014d0f0a5d: exit status 128:
                fatal: could not read Username for 'https://github.com': terminal prompts disabled
        Confirm the import path was entered correctly.
        If this is a private repository, see https://golang.org/doc/faq#git_https for additional information.

There should be a problem with the go-micro project. This is a known thing.

Fork sentinel-go-adapters into your own warehouse, delete the micro related ones, then go mod tidy and submit it to your own warehouse.

go get github.com/phprao/[email protected]

import sentinelPlugin "github.com/phprao/sentinel-go-adapters/gin"

Introduced using middleware

r := gin.New()
r.Use(sentinelPlugin.SentinelMiddleware())

This is the default way, and its effect is

  • Resource name: {method}:{path}, such as "GET:/api/users/:id"
  • If current limiting is triggered, 429 (http.StatusTooManyRequests) status code will be returned.
  • The default behavior can be changed by passing in Option

method signature

// SentinelMiddleware returns new gin.HandlerFunc
// Default resource name is {method}:{path}, such as "GET:/api/users/:id"
// Default block fallback is returning 429 code
// Define your own behavior by setting options
func SentinelMiddleware(opts ...Option) gin.HandlerFunc {<!-- -->
options := evaluateOptions(opts)
return func(c *gin.Context) {<!-- -->
resourceName := c.Request.Method + ":" + c.FullPath()

if options.resourceExtract != nil {<!-- -->
resourceName = options.resourceExtract(c)
}
// bury the point
entry, err := sentinel.Entry(
resourceName,
sentinel.WithResourceType(base.ResTypeWeb),// Mark the classification of the hidden resource
sentinel.WithTrafficType(base.Inbound),//Inbound traffic
)

         // Current limiting is triggered
if err != nil {<!-- -->
if options.blockFallback != nil {<!-- -->
options.blockFallback(c)
} else {<!-- -->
c.AbortWithStatus(http.StatusTooManyRequests)
}
return
}
//Be sure to close the entry after the business logic is completed
defer entry.Exit()
c.Next()
}
}

Two Option are provided, and Entry only encapsulates these two.

// WithResourceExtractor sets the resource extractor of the web requests.
func WithResourceExtractor(fn func(*gin.Context) string) Option {<!-- -->
return func(opts *options) {<!-- -->
opts.resourceExtract = fn
}
}

// WithBlockFallback sets the fallback handler when requests are blocked.
func WithBlockFallback(fn func(ctx *gin.Context)) Option {<!-- -->
return func(opts *options) {<!-- -->
opts.blockFallback = fn
}
}

They are used to set resourceName and return information after triggering current limiting respectively. If resourceName is set to the client IP, it becomes IP current limiting, for example.

// customize resource extractor if required
// method_path by default
WithResourceExtractor(func(ctx *gin.Context) string {<!-- -->
    return ctx.GetHeader("X-Real-IP")
}),

// customize block fallback if required
// abort with status 429 by default
WithBlockFallback(func(ctx *gin.Context) {<!-- -->
    ctx.AbortWithStatusJSON(400, map[string]interface{<!-- -->}{<!-- -->
        "err": "too many requests; the quota used up",
        "code": 10222,
    })
}),

The rest is initialization

import sentinel "github.com/alibaba/sentinel-golang/api"

err := sentinel.InitDefault()
if err != nil {<!-- -->
    t.Fatalf("Unexpected error: % + v", err)
}

Then load the flow control rules

import "github.com/alibaba/sentinel-golang/core/flow"

_, err = flow.LoadRules([]*flow.Rule{<!-- -->
{<!-- -->
Resource: "GET:/show/stats",
Threshold: 10, // 10 per second
TokenCalculateStrategy: flow.Direct,// means directly using the field Threshold as the threshold
ControlBehavior: flow.Reject,//Indicates direct rejection if the threshold is exceeded
         StatIntervalInMs: 1000,//Statistical period, in milliseconds, setting it to 1000 means QPS
},
})
if err != nil {<!-- -->
//Failed to load rules, perform related processing
}

test

func TestSentinel(t *testing.T) {<!-- -->
for i := 0; i < 20; i + + {<!-- -->
t.Run("run-" + strconv.Itoa(i), func(t *testing.T) {<!-- -->
resp, err := http.Get("http://127.0.0.1:8007/show/stats")
if err != nil {<!-- -->
log.Println(err)
} else {<!-- -->
_, err := io.ReadAll(resp.Body)
if err != nil {<!-- -->
log.Println(resp.StatusCode, err)
} else {<!-- -->
log.Println(resp.StatusCode)
}
}

})
}
}

=== RUN TestSentinel/run-0
2023/09/27 14:57:50 200
=== RUN TestSentinel/run-1
2023/09/27 14:57:50 200
=== RUN TestSentinel/run-2
2023/09/27 14:57:50 200
=== RUN TestSentinel/run-3
2023/09/27 14:57:50 200
=== RUN TestSentinel/run-4
2023/09/27 14:57:50 200
=== RUN TestSentinel/run-5
2023/09/27 14:57:50 200
=== RUN TestSentinel/run-6
2023/09/27 14:57:50 200
=== RUN TestSentinel/run-7
2023/09/27 14:57:50 200
=== RUN TestSentinel/run-8
2023/09/27 14:57:50 200
=== RUN TestSentinel/run-9
2023/09/27 14:57:50 200
=== RUN TestSentinel/run-10
2023/09/27 14:57:50 429
=== RUN TestSentinel/run-11
2023/09/27 14:57:50 429
=== RUN TestSentinel/run-12
2023/09/27 14:57:50 429
=== RUN TestSentinel/run-13
2023/09/27 14:57:50 429
=== RUN TestSentinel/run-14
2023/09/27 14:57:50 429
=== RUN TestSentinel/run-15
2023/09/27 14:57:50 429
=== RUN TestSentinel/run-16
2023/09/27 14:57:50 429
=== RUN TestSentinel/run-17
2023/09/27 14:57:50 429
=== RUN TestSentinel/run-18
2023/09/27 14:57:50 429
=== RUN TestSentinel/run-19
2023/09/27 14:57:50 429

dashboard console

Download the jar package: https://github.com/alibaba/Sentinel/releases/download/1.8.6/sentinel-dashboard-1.8.6.jar

> java -Dserver.port=8080 -Dcsp.sentinel.dashboard.server=localhost:8080 -Dproject.name=sentinel-dashboard -jar sentinel-dashboard-1.8.6.jar

Visit: localhost:8080

Login: sentinel/sentinel

I still don’t know how to access golang.