noizeramp

Aleksey's programming, engineering and electronics log

Merging responses in Krakend

Aleksey Gureev
8 August 2022 ⋅ microservices

Krakend is exciting. Among all different features there’s one tech that’s particularly interesting and helpful. It’s possible to define an endpoint that calls several underlying services (either sequentially or concurrently) and merges the responses.

When it’s sequential calling, it even lets you use the data returned from the previous calls in the following. However, at the moment of writing (Krakend 2.0.5) it’s limited to using JSON field values as parts of the request path only. You cannot compose bodies of following requests using this data declaratively.

Basic scenario

Here’s a simple endpoint definition:

{
  "endpoint": {
    "endpoint": "/api/users/{id}",
    "method": "GET",
    "output_encoding": "json",

    "backends": [
      {
        "host": [ "users" ],
        "url_pattern": "/api/users/{id}"
      },

      {
        "host": [ "media" ],
        "url_patthern": "/api/media/{resp0_user_media_id}"
      }
    ]
  }
}

When the client calls GET /api/users/123 Krakend sends the request to the user service for user details:

// GET http://users/api/users/123

{
  "user": {
    "id": 123,
    "photo_id": 567
  }
}

Then uses the photo_id field to query user photo URL from the media service. Assuming that GET /api/media/<id> returns media details as below:

// GET http://media/api/media/567

{
  "media": {
    "id": 567,
    "url": "https://example.com/media/image-567.jpg"
  }
}

Finally, it merges two responses and the result looks like this:

{
  "user": {
    "id": 123,
    "photo_id": 567
  },

  "media": {
    "id": 567,
    "url": "https://example.com/media/image-567.jpg"
  }
}

Complex scenario

What if we get the list of users and get data as follows and still want to list media file details for the mentioned photo_id values?

{
  "data": [
    { "id": 111, "photo_id": 456 },
    { "id": 222, "photo_id": null },
    { "id": 333, "photo_id": 789 },
  ]
}

Lua scripting to the rescue. One option is to collect all non-null values of photo_id fields from the first response and make the call to the media server if there are any; then parse the response from it and inject.

{
  "endpoint": "/api/users",
  "method": "GET",
  "output_encoding": "json",
  "backend": [
    {
      "url_pattern": "/api/users",
      "host": [ "users" ],
      "extra_config": {
        "modifier/lua-backend": {
          "sources": [
            "config/sources/libs/json.lua",
            "config/sources/media.lua",
            "config/sources/users.lua"
          ],
          "post": "inject_list_photos(request.load(), response.load(), http_response, 'media')",
          "allow_open_libs": true
        }
      }
    }
  ],
}

Things to note:

  1. output_encoding is set to json. It tells Krakend to parse the response from the first call and provide it to the script as a Lua table. If it no-op, parsing will not happen and only the raw text body will be available.
  2. sources contains the list of Lua script files to load. Functions that are used in pre / post values can be defined there. In addition, I put a JSON parsing / stringifying library there too. Krakend limits the number of things you can use in scripting environment to a bare minimun (and of course, no external libs).
  3. post is called (yes, you’ve guessed it) after the request to the service is completed and successful.
  4. allow_open_libs lets us use some standard constructs in the script.
  5. http_response is a Krakend helper that’s purpose is to let you make nested HTTP(S) requests.

Now assume that we have an endpoint that returns info for multiple media IDs:

// GET http://media/api/media?ids=1,2

{
  "media": [
    {
      "id": 1,
      "url": "https://example.com/media/image-1.jpg"
    },
    {
      "id": 2,
      "url": "https://example.com/media/image-2.jpg"
    }
  ]
}

We can write a small Lua script to use that:

-- config/sources/users.lua

-- Walks through the users list and picks all non-null photo_id values
function collect_photo_ids(users)
  local photo_ids = {}

  for i = 0, users:len() - 1, 1 do
    local emp = users:get(i)
    local photo_id = emp:get("photo_id")

    if photo_id then
      table.insert(photo_ids, photo_id)
    end
  end

  return photo_ids
end

-- Requests media details for the collected ids
function request_media_info(ids, httpResponse, request, media_host)
  if #ids == 0 then
    return {}
  end

  -- GET https://media/api/media?ids=1,2,3,4
  local r = httpResponse.new("http://" .. media_host .. "/api/media?ids=" .. table.concat(ids, ','), "GET")

  local statusCode = r:statusCode()
  local body = r:body()
  r:close()

  if statusCode ~= 200 then
    custom_error("Failed to fetch media info")
  end

  -- json_parse function is coming from config/sources/lib/json.lua
  -- Originally coming from https://gist.github.com/tylerneylon/59f4bcf316be525b30ab with a slight chage
  -- or replacing `json.parse` with `json_parse`
  return json_parse(body)
end

-- Injects photo details into `photos` key of the first query response
function inject_list_photos(request, resp, httpResponse, media_host)
  local body = resp:data()
  local employees = body:get("data")

  local photo_ids = collect_photo_ids(employees)

  body:set("photos", request_media_info(photo_ids, httpResponse, request, media_host))
end


Comments