One problem you can encounter on the Fediverse is federation problems between servers. Sometimes these are bugs, sometimes it’s just because certain activities aren’t implemented yet. Since these issues happen between different instances and sometimes between different software, this whole federation thing may feel a bit like black magic at times. I wanted to have a better understanding on how to debug federation issues on Pleroma, so decided to get some insight into how federation in Pleroma works. This article is written in the assumption that you follow along while actually doing the steps I’ve done. I don’t think simply reading over it will help you much, but who knows. I mostly hope this article can help people debug problems when they see it and make it easier to address the problems when they arise.
There are two ways to get things federated to Pleroma. The remote instance can POST to the inbox of your instance, and you can search the remote object in the search field of your instance.
We’ll take a closer look into both of them.
POST from a remote server
Let’s start with posting to the inbox. When a remote instance sends something to your server, the first question is if your server receives the request. If you use nginx, you can see the incoming requests in your access_log. You can set the location in the server block of your nginx config file. If you have multiple services running, it’s best to use a separate access_log when debugging. In the server block I see that the access_log I have for the instance I’m using here is /var/log/nginx/develop.ilja.space-access.log. I follow the access_log file with tail -f /var/log/nginx/develop.ilja.space-access.log. tail shows the last few lines of the file, the -f option is to “follow” the file, meaning you’ll see the output updating in real time. To exit this, press ctrl+c. if you change your nginx files to get a separate access_log, don’t forget to run systemctl reload nginx.
Once this is set, you can try to send a post from a remote server that your server should receive. To do this, I’ve set up a Plume instance on develop.blog.ilja.space, made a user, looked up the user in Pleroma (develop.ilja.space), followed it, and then posted a new article in Plume. In my access_log I see
213.219.144.83 - - [13/Sep/2021:10:49:48 +0200] "POST /inbox HTTP/1.1" 200 4 "-" "Plume/0.6.0"
Ok, we see the request comming in, but it doesn’t give us much information. The next question is how the body looks. You can change the output to the access log in your nginx config. Add the line log_format postdata escape=none '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent $request_body "$http_referer" "$http_user_agent"'; above the server block. Then, in the server block, you change the line for the output to access_log /var/log/nginx/develop.ilja.space-access.log postdata; and load the new settings systemctl reload nginx. This will give us much more information about the actual response.
The following is what we log for a new article.
2001:913:1fff:ffff::9:3318 - [10/Oct/2021:09:14:49 +0200] "POST /inbox HTTP/1.1" 200 4 {"@context":["https://www.w3.org/ns/activitystreams","https://w3id.org/security/v1",{"Emoji":"toot:Emoji","Hashtag":"as:Hashtag","atomUri":"ostatus:atomUri","conversation":"ostatus:conversation","featured":"toot:featured","focalPoint":{"@container":"@list","@id":"toot:focalPoint"},"inReplyToAtomUri":"ostatus:inReplyToAtomUri","manuallyApprovesFollowers":"as:manuallyApprovesFollowers","movedTo":"as:movedTo","ostatus":"http://ostatus.org#","sensitive":"as:sensitive","toot":"http://joinmastodon.org/ns#"}],"actor":"https://develop.blog.ilja.space/@/ilja/","cc":["https://develop.ilja.space/users/ilja"],"id":"https://develop.blog.ilja.space/~/Blabla/some-post/activity","object":{"attributedTo":["https://develop.blog.ilja.space/@/ilja/","https://develop.blog.ilja.space/~/Blabla/"],"cc":["https://develop.ilja.space/users/ilja"],"content":"<p>This is some text. Let’s see how it looks, eh.</p>\n","id":"https://develop.blog.ilja.space/~/Blabla/some-post/","license":"CC BY-SA 4.0","name":"Some post","published":"2021-10-10T07:14:48.661382Z","source":{"content":"This is some text. Let's see how it looks, eh.","mediaType":"text/markdown"},"summary":"to test things out","tag":[{"href":"https://develop.blog.ilja.space/tag/test","name":"test","type":"Hashtag"}],"to":["https://www.w3.org/ns/activitystreams#Public"],"type":"Article","url":"https://develop.blog.ilja.space/~/Blabla/some-post/"},"signature":{"created":"2021-10-10T07:14:49.100509621+00:00","creator":"https://develop.blog.ilja.space/@/ilja/#main-key","signatureValue":"Vt9DllSnQpfqId3F6stKiiuK/whskQftiUsExRhJikWxULrKPFlxsBs0TGX1H7vwBsPR/Vn4LUTpcseEf/KknWhshuwuchVUTA1wwxRAO4mc3yN0mKK8pZuvIxcJ+EnDXOFXJR/vVFHlTkWclZlcZ8YSYJr4Sg1NO/2W5f84ApgdGYL3nSH6cnU4EOOg+yK7PIMWjG3AsXAQFs6DUTP67DW4DUXQa36laI5mMtRUtJXlhusQaLdjaF8QMOMtX66nOhMSZmb2uE7kMZP+lTVDpV6UCiFZmvqv5/luGE6qYnUX/8/lpa4gLEhZMdDWCdX+SXbDo7J6K+t4lMpvuistlg==","type":"RsaSignature2017"},"to":["https://www.w3.org/ns/activitystreams#Public"],"type":"Create"} "" "Plume/0.6.0"
You may have a text editor that can help you make this prettier, but I’ll use a bash command for that since that’s something everyone should be able to do without having to install extra software. In the following, you change the placeholder <BODY> with the body we have and run it. (The body here is the whole curly brackets thing {"@context": ... "type":"Create"}).
cat << EOF | python3 -mjson.tool
<BODY>
EOF
This gives us the response nicely formatted and readable
{
"@context": [
"https://www.w3.org/ns/activitystreams",
"https://w3id.org/security/v1",
{
"Emoji": "toot:Emoji",
"Hashtag": "as:Hashtag",
"atomUri": "ostatus:atomUri",
"conversation": "ostatus:conversation",
"featured": "toot:featured",
"focalPoint": {
"@container": "@list",
"@id": "toot:focalPoint"
},
"inReplyToAtomUri": "ostatus:inReplyToAtomUri",
"manuallyApprovesFollowers": "as:manuallyApprovesFollowers",
"movedTo": "as:movedTo",
"ostatus": "http://ostatus.org#",
"sensitive": "as:sensitive",
"toot": "http://joinmastodon.org/ns#"
}
],
"actor": "https://develop.blog.ilja.space/@/ilja/",
"cc": [
"https://develop.ilja.space/users/ilja"
],
"id": "https://develop.blog.ilja.space/~/Blabla/some-post/activity",
"object": {
"attributedTo": [
"https://develop.blog.ilja.space/@/ilja/",
"https://develop.blog.ilja.space/~/Blabla/"
],
"cc": [
"https://develop.ilja.space/users/ilja"
],
"content": "<p>This is some text. Let\u2019s see how it looks, eh.</p>\n",
"id": "https://develop.blog.ilja.space/~/Blabla/some-post/",
"license": "CC BY-SA 4.0",
"name": "Some post",
"published": "2021-10-10T07:14:48.661382Z",
"source": {
"content": "This is some text. Let's see how it looks, eh.",
"mediaType": "text/markdown"
},
"summary": "to test things out",
"tag": [
{
"href": "https://develop.blog.ilja.space/tag/test",
"name": "test",
"type": "Hashtag"
}
],
"to": [
"https://www.w3.org/ns/activitystreams#Public"
],
"type": "Article",
"url": "https://develop.blog.ilja.space/~/Blabla/some-post/"
},
"signature": {
"created": "2021-10-10T07:14:49.100509621+00:00",
"creator": "https://develop.blog.ilja.space/@/ilja/#main-key",
"signatureValue": "Vt9DllSnQpfqId3F6stKiiuK/whskQftiUsExRhJikWxULrKPFlxsBs0TGX1H7vwBsPR/Vn4LUTpcseEf/KknWhshuwuchVUTA1wwxRAO4mc3yN0mKK8pZuvIxcJ+EnDXOFXJR/vVFHlTkWclZlcZ8YSYJr4Sg1NO/2W5f84ApgdGYL3nSH6cnU4EOOg+yK7PIMWjG3AsXAQFs6DUTP67DW4DUXQa36laI5mMtRUtJXlhusQaLdjaF8QMOMtX66nOhMSZmb2uE7kMZP+lTVDpV6UCiFZmvqv5/luGE6qYnUX/8/lpa4gLEhZMdDWCdX+SXbDo7J6K+t4lMpvuistlg==",
"type": "RsaSignature2017"
},
"to": [
"https://www.w3.org/ns/activitystreams#Public"
],
"type": "Create"
}
Once we hit Pleroma
Now that we know how the request and body looks like, let’s see how Pleroma handles this. We can search the endpoint in the code. I open a terminal in Pleroma/lib/Pleroma and run grep -R '/inbox'. This gives us several results, one being web/router.ex: post("/inbox", ActivityPubController, :inbox). You can look up the code, and that’s probably the smartest thing to do, but if you know a bit of Phoenix, this actually tells us everything we need already. There’s a module called ActivityPubController which has a function inbox. That function is called to further process the request. When we look for that module, grep -R 'ActivityPubController do', we find web/activity_pub/activity_pub_controller.ex:defmodule Pleroma.Web.ActivityPub.ActivityPubController do. This is the definition of this module and thus the file we are looking for. The inbox function looks as follows.
def inbox(%{assigns: %{valid_signature: true}} = conn, %{"nickname" => nickname} = params) do
with %User{} = recipient <- User.get_cached_by_nickname(nickname),
{:ok, %User{} = actor} <- User.get_or_fetch_by_ap_id(params["actor"]),
true <- Utils.recipient_in_message(recipient, actor, params),
params <- Utils.maybe_splice_recipient(recipient.ap_id, params) do
Federator.incoming_ap_doc(params)
json(conn, "ok")
end
end
def inbox(%{assigns: %{valid_signature: true}} = conn, params) do
Federator.incoming_ap_doc(params)
json(conn, "ok")
end
# POST /relay/inbox -or- POST /internal/fetch/inbox
def inbox(conn, params) do
if params["type"] == "Create" && FederatingPlug.federating?() do
post_inbox_relayed_create(conn, params)
else
post_inbox_fallback(conn, params)
end
end
From here you can try to follow what happens. Elixir allows you to debug code, but that requires some setup and if there are timeouts set for the requests, it’s possible that it will time-out before you can see something useful. At least that’s my experience with it. You can also print out things using IO.inspect("thing to print out"). This function can also print out maps and lists. It has a default limit in how big the output can be, but this can be changed by providing a parameter. You can also have it print a label, so you can easily tell in the output where you printed something. IO.inspect("thing to print out", label: "Thing I want to see", limit: :infinity) will print a label and doesn’t have a limit on how big something can be. In Elixir you can also pipe functions. The equivalent would be "thing to print out" |> IO.inspect(label: "Thing I want to see", limit: :infinity). Using IO.inspect isn’t the most efficient way to debug, but it’s the one with the lowest threshold to get started with, so I’ll use that here.
def inbox(%{assigns: %{valid_signature: true}} = conn, %{"nickname" => nickname} = params) do
params |> IO.inspect(label: "inbox 1", limit: :infinity)
with %User{} = recipient <- User.get_cached_by_nickname(nickname),
{:ok, %User{} = actor} <- User.get_or_fetch_by_ap_id(params["actor"]),
true <- Utils.recipient_in_message(recipient, actor, params),
params <- Utils.maybe_splice_recipient(recipient.ap_id, params) do
Federator.incoming_ap_doc(params)
json(conn, "ok")
end
end
def inbox(%{assigns: %{valid_signature: true}} = conn, params) do
params |> IO.inspect(label: "inbox 2", limit: :infinity)
Federator.incoming_ap_doc(params)
json(conn, "ok")
end
# POST /relay/inbox -or- POST /internal/fetch/inbox
def inbox(conn, params) do
params |> IO.inspect(label: "inbox 3", limit: :infinity)
if params["type"] == "Create" && FederatingPlug.federating?() do
post_inbox_relayed_create(conn, params)
else
post_inbox_fallback(conn, params)
end
end
Here’s what came out of the elixir output (you can run with su Pleroma -s $SHELL -c 'MIX_ENV=prod mix phx.server' instead of systemctl start Pleroma, or you can check using journalctl -u Pleroma -f where Pleroma is the user you used for the instance you’re testing on.)
journalctl:
inbox 2: %{
"@context" => [
"https://www.w3.org/ns/activitystreams",
"https://w3id.org/security/v1",
%{
"Emoji" => "toot:Emoji",
"Hashtag" => "as:Hashtag",
"atomUri" => "ostatus:atomUri",
"conversation" => "ostatus:conversation",
"featured" => "toot:featured",
"focalPoint" => %{"@container" => "@list", "@id" => "toot:focalPoint"},
"inReplyToAtomUri" => "ostatus:inReplyToAtomUri",
"manuallyApprovesFollowers" => "as:manuallyApprovesFollowers",
"movedTo" => "as:movedTo",
"ostatus" => "http://ostatus.org#",
"sensitive" => "as:sensitive",
"toot" => "http://joinmastodon.org/ns#"
}
],
"actor" => "https://develop.blog.ilja.space/@/ilja/",
"cc" => ["https://develop.ilja.space/users/ilja"],
"id" => "https://develop.blog.ilja.space/~/Blabla/some-post/activity",
"object" => %{
"attributedTo" => ["https://develop.blog.ilja.space/@/ilja/",
"https://develop.blog.ilja.space/~/Blabla/"],
"cc" => ["https://develop.ilja.space/users/ilja"],
"content" => "<p>This is some text. Let's see how it looks, eh.</p>\n",
"id" => "https://develop.blog.ilja.space/~/Blabla/some-post/",
"license" => "CC BY-SA 4.0",
"name" => "Some post",
"published" => "2021-10-10T07:14:48.661382Z",
"source" => %{
"content" => "This is some text. Let's see how it looks, eh.",
"mediaType" => "text/markdown"
},
"summary" => "to test things out",
"tag" => [%{"href" => "https://develop.blog.ilja.space/tag/test", "name" => "test", "type" => "Hashtag"}],
"to" => ["https://www.w3.org/ns/activitystreams#Public"],
"type" => "Article",
"url" => "https://develop.blog.ilja.space/~/Blabla/some-post/"
},
"signature" => %{
"created" => "2021-10-10T07:26:54.767938546+00:00",
"creator" => "https://develop.blog.ilja.space/@/ilja/#main-key",
"signatureValue" => "ztTa3ZbQICfX1weVR0u2XJlPBrNHlMMmgf3DRDpoKEsd3imHE7OLO1RUm4lds590k9gRkxI7512lALyfPX4D/Z9ck+whl25PpTEqIKuRf7yKYIaA787BTSnlOdpZQKoyyUEYNmOKlB9JtWWNZR5X9ZQhBVhVd6n+QhZWGasNVTHPvx/BHbYD2gOlGyRmUSFqCp7XXrKfwUSvS1xzqwi+jn/uh+/OH9DYlFaUlLO4WsvXx6EZVeSkM8cGlPWRTBbCT1gbGYczYiZqtDBNiyzXoORGyd9zPpLfLtSWO7LcPu04CJAVDK4zbyuJJUkykvz94ipswYROmiOaB71aPNIwVA==",
"type" => "RsaSignature2017"
},
"to" => ["https://www.w3.org/ns/activitystreams#Public"],
"type" => "Create"
}
There’s not really much surprise in how it looks when we compare it to the body nginx logged, but the label inbox 2 does tell us what function is called. Let’s check Federator.incoming_ap_doc(params).
After grepping and following the code, we see that it’s put in a queue where it’s handled by calling Federator.perform, which looks like
def perform(:incoming_ap_doc, params) do
Logger.debug("Handling incoming AP activity")
actor =
params
|> Map.get("actor")
|> Utils.get_ap_id()
# NOTE: we use the actor ID to do the containment, this is fine because an
# actor shouldn't be acting on objects outside their own AP server.
with {_, {:ok, _user}} <- {:actor, ap_enabled_actor(actor)},
nil <- Activity.normalize(params["id"]),
{_, :ok} <-
{:correct_origin?, Containment.contain_origin_from_id(actor, params)},
{:ok, activity} <- Transmogrifier.handle_incoming(params) do
{:ok, activity}
else
{:correct_origin?, _} ->
Logger.debug("Origin containment failure for #{params["id"]}")
{:error, :origin_containment_failed}
%Activity{} ->
Logger.debug("Already had #{params["id"]}")
{:error, :already_present}
{:actor, e} ->
Logger.debug("Unhandled actor #{actor}, #{inspect(e)}")
{:error, e}
{:error, {:validate_object, _}} = e ->
Logger.error("Incoming AP doc validation error: #{inspect(e)}")
Logger.debug(Jason.encode!(params, pretty: true))
e
e ->
# Just drop those for now
Logger.debug(fn -> "Unhandled activity\n" <> Jason.encode!(params, pretty: true) end)
{:error, e}
end
end
The with in elixir is a way to try to match several things in a row, and when they do, do something. When something fails, we immediately go to the else branch.
First we do {_, {:ok, _user}} <- {:actor, ap_enabled_actor(actor)}, which means that we expect ap_enabled_actor(actor) to return {:ok, _user}. The _user is matches with everything, so the value there can be a map or list or string or whatever. The underscore is a way to tell elixir that we won’t use this parameter further. Basically, we’re just checking if the actor is an existing actor. When the return value doesn’t match, it will match in the else branch where the errors are handled.
A couple of other things are also checked and you can see in the logs if errors come out of this. Just note that you’ll need to set the loglevel to DEBUG for some of the errors.
The last and most important one is Transmogrifier.handle_incoming(params). The transmogrifier changes activities into something Pleroma understands internally and every object Pleroma can handle should pass here. If it fails, it’s either an object Pleroma doesn’t understand yet, or there’s something really wrong with the incoming object. Whether it’s something that needs changing in the transmogrifier or in the software of the remote instance should be checked on a case-by-case basis.
The harder part is figuring out what function is used. You can either try to follow by sending a new activity, or try to figure out for yourself what’s the first function that will match.
Here we have an type Create, so the following is probably what will match:
def handle_incoming(
%{"type" => "Create", "object" => %{"type" => objtype, "id" => obj_id}} = data,
options
)
when objtype in ~w{Question Answer ChatMessage Audio Video Event Article Note Page} do
fetch_options = Keyword.put(options, :depth, (options[:depth] || 0) + 1)
object =
data["object"]
|> strip_internal_fields()
|> fix_type(fetch_options)
|> fix_in_reply_to(fetch_options)
data = Map.put(data, "object", object)
options = Keyword.put(options, :local, false)
with {:ok, %User{}} <- ObjectValidator.fetch_actor(data),
nil <- Activity.get_create_by_object_ap_id(obj_id),
{:ok, activity, _} <- Pipeline.common_pipeline(data, options) do
{:ok, activity}
else
%Activity{} = activity -> {:ok, activity}
e -> e
end
end
After doing some transformations and extra checks, we call Pipeline.common_pipeline(data, options). Here we store things and federate. I’m not going to go deeper into everything that happens here, because I’m not sure myself, but you can check it out if you need. One thing this function does, is calling do_common_pipeline where some general things happen, including passing the message through the enabled mrf’s. One of the possible MRF’s is DropPolicy which will drop all incoming activity it sees and log the activity in DEBUG mode. This can be helpful in case of debugging.
def do_common_pipeline(message, meta) do
with {_, {:ok, message, meta}} <- {:validate, object_validator().validate(message, meta)},
{_, {:ok, message, meta}} <- {:mrf, mrf().pipeline_filter(message, meta)},
{_, {:ok, message, meta}} <- {:persist, activity_pub().persist(message, meta)},
{_, {:ok, message, meta}} <- {:side_effects, side_effects().handle(message, meta)},
{_, {:ok, _}} <- {:federation, maybe_federate(message, meta)} do
{:ok, message, meta}
else
{:mrf, {:reject, message, _}} -> {:reject, message}
e -> {:error, e}
end
end
Fetching from a remote server
Another way to federate things to your instance, is to fetch them. You can put the URL of posts or users in the search bar and Pleroma will fetch them. But what if Pleroma doesn’t find anything, even though you’re sure you used the correct URL?
When you search in Pleroma, it will look in it’s own database, but it can also fetch objects from the remote instance when you use an URL or a username of the form [email protected]. Let’s see how that happens.
For this I use a second Pleroma instance. I have a post on the server where I’m checking nginx logs and search that post from the other server (who doesn’t know about this post yet). I also enable DropPolicy so I can do the same search multiple times. The URL I search in the search bar is https://develop.ilja.space/notice/AC0oc6vJaByaG2rLTk. I get the following output in the access_log:
80.67.181.196 - [10/Oct/2021:08:41:21 +0200] "GET /notice/AC0oc6vJaByaG2rLTk HTTP/1.1" 302 137 "" "Pleroma 2.4.51-139-gd8d819dd-develop; https://test.pl.ilja.space <[email protected]>"
80.67.181.196 - [10/Oct/2021:08:41:21 +0200] "GET /notice/AC0oc6vJaByaG2rLTk HTTP/1.1" 302 137 "" "Pleroma 2.4.51-139-gd8d819dd-develop; https://test.pl.ilja.space <[email protected]>"
80.67.181.196 - [10/Oct/2021:08:41:21 +0200] "GET /objects/4a9a7f59-c969-4bc7-88af-f243e5d4715e HTTP/1.1" 200 887 "" "Pleroma 2.4.51-139-gd8d819dd-develop; https://test.pl.ilja.space <[email protected]>"
80.67.181.196 - [10/Oct/2021:08:41:21 +0200] "GET /objects/4a9a7f59-c969-4bc7-88af-f243e5d4715e HTTP/1.1" 200 887 "" "Pleroma 2.4.51-139-gd8d819dd-develop; https://test.pl.ilja.space <[email protected]>"
80.67.181.196 - [10/Oct/2021:08:41:21 +0200] "GET /users/test_fetch_user HTTP/1.1" 200 1863 "" "Pleroma 2.4.51-139-gd8d819dd-develop; https://test.pl.ilja.space <[email protected]>"
Now we know what endpoints are called. It seems it starts with a GET request to /notice/AC0oc6vJaByaG2rLTk, which returns a 302, after which we GET /objects/4a9a7f59-c969-4bc7-88af-f243e5d4715e and /users/test_fetch_user. I’m not 100% sure why it tries to fetch the object twice, but possibly it’s because the search tries to look for objects as well as actors, and the code does this as separate things. For the rest we can make a good guess at what happens. We fetch the /notice/AC0oc6vJaByaG2rLTk, the server tells us we should fetch /objects/4a9a7f59-c969-4bc7-88af-f243e5d4715e instead. When we process the object, we don’t find the actor (in this case the user) in our own database, so we fetch that as well. Let’s see if we can use curl to reproduce these steps. Note that we’ll have to tell the server what answer we want to accept, I also use the -v flag to get more verbose output.
curl -v 'https://develop.ilja.space/notice/AC0oc6vJaByaG2rLTk' -H 'Content-Type: application/json; charset=utf-8' -H 'Accept: application/ld+json; profile="https://www.w3.org/ns/activitystreams"'
We see that we indeed get a 302 with location https://develop.ilja.space/objects/4a9a7f59-c969-4bc7-88af-f243e5d4715e which we can also try to fetch. We can also try to fetch the user.
curl -v 'https://develop.ilja.space/objects/4a9a7f59-c969-4bc7-88af-f243e5d4715e' -H 'Content-Type: application/json; charset=utf-8' -H 'Accept: application/ld+json; profile="https://www.w3.org/ns/activitystreams"'
# Which gives us the following body
{"@context":["https://www.w3.org/ns/activitystreams","https://develop.ilja.space/schemas/litepub-0.1.jsonld",{"@language":"und"}],"actor":"https://develop.ilja.space/users/test_fetch_user","attachment":[],"attributedTo":"https://develop.ilja.space/users/test_fetch_user","cc":["https://develop.ilja.space/users/test_fetch_user/followers"],"content":"A post we'll search for from another instance","context":"https://develop.ilja.space/contexts/e13056d4-b83a-429b-81af-fce9599a3c73","conversation":"https://develop.ilja.space/contexts/e13056d4-b83a-429b-81af-fce9599a3c73","id":"https://develop.ilja.space/objects/4a9a7f59-c969-4bc7-88af-f243e5d4715e","published":"2021-10-04T06:15:07.667772Z","repliesCount":1,"sensitive":null,"source":"A post we'll search for from another instance","summary":"","tag":[],"to":["https://www.w3.org/ns/activitystreams#Public"],"type":"Note"}
curl -v 'https://develop.ilja.space/users/test_fetch_user' -H 'Content-Type: application/json; charset=utf-8' -H 'Accept: application/ld+json; profile="https://www.w3.org/ns/activitystreams"'
# Which gives us the following body
{"@context":["https://www.w3.org/ns/activitystreams","https://develop.ilja.space/schemas/litepub-0.1.jsonld",{"@language":"und"}],"alsoKnownAs":[],"attachment":[],"capabilities":{"acceptsChatMessages":true},"discoverable":false,"endpoints":{"oauthAuthorizationEndpoint":"https://develop.ilja.space/oauth/authorize","oauthRegistrationEndpoint":"https://develop.ilja.space/api/v1/apps","oauthTokenEndpoint":"https://develop.ilja.space/oauth/token","sharedInbox":"https://develop.ilja.space/inbox","uploadMedia":"https://develop.ilja.space/api/ap/upload_media"},"featured":"https://develop.ilja.space/users/test_fetch_user/collections/featured","followers":"https://develop.ilja.space/users/test_fetch_user/followers","following":"https://develop.ilja.space/users/test_fetch_user/following","id":"https://develop.ilja.space/users/test_fetch_user","inbox":"https://develop.ilja.space/users/test_fetch_user/inbox","manuallyApprovesFollowers":false,"name":"test_fetch_user","outbox":"http* Connection #0 to host develop.ilja.space left intact
s://develop.ilja.space/users/test_fetch_user/outbox","preferredUsername":"test_fetch_user","publicKey":{"id":"https://develop.ilja.space/users/test_fetch_user#main-key","owner":"https://develop.ilja.space/users/test_fetch_user","publicKeyPem":"-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAvDZWVSVBH2OZLMSxLu7h\nILfYlcLXblFh+EPA4rSpAHYdD3jh2kXR/c+0Dq80/RSJ7ZgOCn2BnS+ShvhgWbI9\nK49VYWZGCOojuSa4SHwch5fJOH3K7zVcMX+1VJF83vi8iEIEkpc5br7UHXhv5TXK\n6Bk+F1BMFWUN5FZ3crhTNf8Yn0Gd9bVVoasfCFgXCLykGiQra03F897K64N2U2Qx\njL2FdSJN2F0O5p2pjoZvpcvVqfb7JXDKs/SbvPrYUHvPOUf7KJhXwuUqzO7cH7Lr\nhFSpZrUj9dwxH6H9NMKxkI+wIxsEA/LU8TmWjx8IXao7JDtN2mAwlK0/Xapj1Slm\nWQIDAQAB\n-----END PUBLIC KEY-----\n\n"},"summary":"","tag":[],"type":"Person","url":"https://develop.ilja.space/users/test_fetch_user"}
To go further in the code, you can set the loglevel to DEBUG and enable DropPolicy. From there you’ll have to dig deeper in a similar way we did before.
Conclusion
When you want to check for incoming AP requests, you may first want to reproduce the problem using a server that doesn’t get much traffic. When you enable better logging in your access_log file and change the log to DEBUG, you can already see a lot. You know if the activity hits your server, and if it already gets as far as Federator.perform. From there you can dig deeper. With a bit of luck you’ll already see some other messages that can help you.
Figuring out federation issues still needs some work, but I hope this article makes it a bit clearer how this works in Pleroma. Note that this was written when Pleroma 2.4.1 was the latest version. It’s possible that certain flows have changed over time. Nevertheless, I hope this article contains enough info for you to figure out the flow yourself if needed.
If you have feedback, or things that you think are wrong or could be done better, feel free to let me know.
Good luck!
Edits
- 2021-10-11: I used
application/jsonfor the accept header because that worked with Pleroma. I later tried with Misskey, who returned the html page instead of the json object. I looked into it, found the correct way to do it in the AP spec, and changed the examples accordingly.
Comments
No comments yet. Be the first to react!