Rill Flow

Overview

Rill Flow is a high-performance and scalable distributed workflow orchestration service, originally designed to tackle the complexities and high concurrency demands of video processing workflows in the Weibo video business context. Key features of Rill Flow include user-friendliness, high concurrency, and low latency, which are essential for efficient task management. Currently, Rill Flow is extensively utilized in Weibo's video publishing and transcoding operations, handling tens of millions of tasks daily and supporting core processes of the video services.

Additionally, with the widespread adoption of Large Language Models (LLMs), Rill Flow has undergone optimizations based on cloud-native architecture and adapted for LLMs through plugin integration. This enhancement significantly improves scalability, enabling swift support for AI-generated content (AIGC) workflow integration.

QuickStart

Tool Preparation

Before you begin, ensure that the following tools are installed:

• Environment suitable for OSX/Linux
• Docker
• Docker-Compose

Service Deployment

Install Rill Flow services on your local environment using Docker-Compose:

cat << EOF > docker-compose.yaml
version: '3'
services:
  rill-flow:
    image: weibocom/rill-flow
    depends_on:
      - cache
      - jaeger
    ports:
      - "8080:8080"
    environment:
      - RILL_FLOW_DESCRIPTOR_REDIS_HOST=cache
      - RILL_FLOW_DEFAULT_REDIS_HOST=cache
      - RILL_FLOW_TRACE_ENDPOINT=http://jaeger:4317
      - RILL_FLOW_CALLBACK_URL=http://rill-flow:8080/flow/finish.json
  cache:
    image: redis:6.2-alpine
    restart: always
    command: redis-server --save 20 1 --loglevel warning
  jaeger:
    image: jaegertracing/all-in-one:1.39
    restart: always
    environment:
      - COLLECTOR_OTLP_ENABLED=true
  ui:
    image: weibocom/rill-flow-ui
    ports:
      - "8088:80"
      - "8089:8089"
    depends_on:
      - rill-flow
      - jaeger
    environment:
      - BACKEND_SERVER=http://rill-flow:8080
      - TRACE_SERVER=http://jaeger:16686
  sample-executor:
    image: weibocom/rill-flow-sample:sample-executor 
EOF
docker-compose up -d

Execution Example

• Step 1: Submit a YAML file defining a workflow

curl --location  --request POST 'http://127.0.0.1:8080/flow/bg/manage/descriptor/add_descriptor.json?business_id=rillFlowSimple&feature_name=greet&alias=release' \
--header 'Content-Type: text/plain' \
--data-raw '---
version: 1.0.0
workspace: rillFlowSimple
dagName: greet
type: flow
tasks:
  - category: function
    name: Bob 
    resourceName: http://sample-executor:8000/greet.json?user=Bob
    pattern: task_sync
    tolerance: false
    next: Alice
    inputMappings:
      - source: "$.context.Bob"
        target: "$.input.Bob"
  - category: function
    name: Alice 
    resourceName: http://sample-executor:8000/greet.json?user=Alice
    pattern: task_sync
    tolerance: false
    inputMappings:
      - source: "$.context.Alice"
        target: "$.input.Alice"
'

• Step 2: Submit a task to execute the workflow

curl -XPOST 'http://127.0.0.1:8080/flow/submit.json?descriptor_id=rillFlowSimple:greet'  -d '{"Bob":"Hello, I am Bob!", "Alice": "Hi, I am Alice"}' -H 'Content-Type:application/json'

• Step 3：Query the task execution results

  Query the execution details via the Rill Flow management console.(admin/admin)

http://127.0.0.1:8088/#/flow-instance/list

Document

https://rill-flow.github.io/en/docs/intro

Contributors

The following are contributors to the project along with their GitHub links:

• axb (@qdaxb)
• Ocean (@hhh041)
• xilong-t (@xilong-t)
• techlog (@techloghub)
• ch15084 (@ch15084)
• qfl (@qiaofenlin)
• Kylen (@Kylen)
• zzfzzf (@zzfzzf)
• feifei (@feifei325)

License

Rill Flow is an open-source project under the Apache License 2.0.