In the first part of the series, I introduced the architecture for building a link sharing relay. This post covers the actual code needed to make that relay work. I’m writing in F#, a beautiful language.
Azure functions provide a simple environment to deploy code, requiring a minimum of development overhead. Nevertheless, Azure Functions are not low-code or no-code solutions. We will need to write some code. For this project, I chose F#, so that I could develop my skills in the language and learn a bit more about functional programming. I also wanted to try out the Railway Pattern, which absolutely blew my mind when I first read about it. It’s an awesome way to write code, and I can’t say I’ve mastered it, but I wanted to try it out.
Posts without posting, a series
To get started, you’re going to need the .Net 6.0 runtime and the Azure Functions Core Tools. I strongly recommend using VSCode with the Ionide extension. I’m developing on Ubuntu 20.04 LTS on Windows System Linux (WSL) 2, but you should be able to do this on Mac, Linux, or Windows, if you prefer.
This will be a long post, so I’ll try to break it down for you. We’ll do the following steps.
run function and configure it to accept POST requestsLet’s begin at the beginning. We’re going to set up an azure function. You can do this wherever you’d like, but I did it in my static site repository. This might have been a mistake, so I’ll recommend you do this in its own folder. You won’t have to create it, the tools should set everything up for you. Assuming you’ve installed the Azure Functions Core Tools linked above, simply type
func init [PROJECT NAME]
Replace [PROJECT NAME] with your project’s name. This should prompt you to select a worker runtime:
Select a number for worker runtime:
1. dotnet
2. dotnet (isolated process)
3. node
4. python
5. powershell
6. custom
Choose option:
Choose Option 1, for dotnet. Option 2 might resolve the issue I was having with .Net 7.0, but I haven’t tested this yet, so let’s stick with what’s known and easy. You’ll next be prompted to pick a language. Choose F#, if you dare.
This should create a project folder for you, initialize it as a git repository, create a .gitignore, and create a handful of files: [PROJECT NAME].csproj, host.json, local.settings.json, and a couple of directories.
Navigate to this file, and let’s initialize a new function. Type func new -n [FUNCTION NAME] -l F# to create a new function [FUNCTION NAME]. This can be the same as your project name, but need not be. This is because a single function app deployment can have multiple endpoints, each pointing to a different function. When prompted, choose HttpTrigger. Running this command should add a couple of new files: [FUNCTION NAME].fs, metadata.json, and function.json.
Let’s go ahead and rename [PROJECT NAME].csproj to [PROJECT NAME].fsproj. I don’t know that it matters, but I feel a little weird calling an F# project like a C# project.
Let’s open the .fsproj file, which should have some XML inside. Find the block that says
<ItemGroup>
<None Update="host.json">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
<None Update="local.settings.json">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<CopyToPublishDirectory>Never</CopyToPublishDirectory>
</None>
</ItemGroup>
We need to do some small changes here. Update the block as follows:
<ItemGroup>
<Compile Include="[FUNCTION NAME].fs" />
<None Include="host.json">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
<None Include="local.settings.json" Condition="Exists('local.settings.json')">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<CopyToPublishDirectory>Never</CopyToPublishDirectory>
</None>
</ItemGroup>
The first change ensures that we compile our F# code and include it in the build. The second changes involve modifying the Update attribute to be Include. This is necessary for local testing. Finally, we add a Condition attribute to further simplify local testing.
Open your .fsproj file, and in the second <ItemGroup> block, just before <None Include="host.json"> add the following snippet: <Compile Include=[FUNCTION NAME].fs" />. This ensures that your code is compiled and built. You’ll need to do this for every file that you add.
In VSCode (or your preferred IDE), open [FUNCTION NAME].fs. It should have an autogenerated template that looks something like this. From now on, I’ll be using LinkSharer as my project name and function name.
namespace LinkSharer
open System
open System.IO
open Microsoft.AspNetCore.Mvc
open Microsoft.Azure.WebJobs
open Microsoft.Azure.WebJobs.Extensions.Http
open Microsoft.AspNetCore.Http
open Newtonsoft.Json
open Microsoft.Extensions.Logging
module LinkSharer =
// Define a nullable container to deserialize into.
[<AllowNullLiteral>]
type NameContainer() =
member val Name = "" with get, set
// For convenience, it's better to have a central place for the literal.
[<Literal>]
let Name = "name"
[<FunctionName("LinkSharer")>]
let run ([<HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)>]req: HttpRequest) (log: ILogger) =
async {
log.LogInformation("F# HTTP trigger function processed a request.")
let nameOpt =
if req.Query.ContainsKey(Name) then
Some(req.Query.[Name].[0])
else
None
use stream = new StreamReader(req.Body)
let! reqBody = stream.ReadToEndAsync() |> Async.AwaitTask
let data = JsonConvert.DeserializeObject<NameContainer>(reqBody)
let name =
match nameOpt with
| Some n -> n
| None ->
match data with
| null -> ""
| nc -> nc.Name
let responseMessage =
if (String.IsNullOrWhiteSpace(name)) then
"This HTTP triggered function executed successfully. Pass a name in the query string or in the request body for a personalized response."
else
"Hello, " + name + ". This HTTP triggered function executed successfully."
return OkObjectResult(responseMessage) :> IActionResult
} |> Async.StartAsTask
This is boilerplate code and it should run locally. Fun func host start and try sending a request with curl to the URL that it displays, typically something like curl http://localhost:7071/api/FUNCTION NAME. You should see a message like, This HTTP triggered function executed successfully. Pass a name in the query string or in the request body for a personalized response. If so, congratulations! We’ve configured our F# function, let’s move onto the next step.
run functionTo build our link sharer, we’re going to strip out a lot of the boilerplate and cruft. This is actually going to be very simple. We’ll start with the run function. In your LinkSharer.fs file, let’s modify it as follows.
[<FunctionName("LinkSharer")>]
let run ([<HttpTrigger(AuthorizationLevel.Function, "post", Route = null)>]req: HttpRequest) (log: ILogger) =
async {
log.LogInformation("F# HTTP trigger function processed a request.")
let! (data: LinkData) = getPostFromReq req
let tweet: Async<Result<ID,string>> =
getTwitterClient
>>@ writeTweet data
let sheet: Async<Result<ID,string>> =
getSheetService
>>@ writeToGoogleSheet data
let toot: Async<Result<ID,string>> =
getMastodonClient
>>@ writeToot data
let! (result: Result<ID,string>) =
tweet
>>== toot
>>== sheet
return match result with
| Success s -> OkObjectResult(s) :> IActionResult
| Failure f -> StatusCodeResult(500) :> IActionResult
} |> Async.StartAsTask
This is actually shockingly simple. First, I’ve stripped off get from the function call, as this will be a POST-only endpoint. Notice AuthorizationLevel.Function, this forces us to authenticate to use the API. We don’t want randos sharing to our social media!
Next, we have let! (data: LinkData) = getPostFromReq req. This takes the JSON payload attached to the request and processes it with a function called getPostFromReq. Essentially, all this does is take the request data and format it using an F# Record type. Let’s create that code now. Above the run method, add the following code.
type LinkData = {
url: string
title: string
comment: string
}
let getPostFromReq (req: HttpRequest) =
async {
use stream: StreamReader = new StreamReader(req.Body)
let! (reqBody: string) = stream.ReadToEndAsync() |> Async.AwaitTask
return JsonConvert.DeserializeObject<LinkData>(reqBody)
}
We probably don’t need the async here, but let’s use it. What this does is reads the JSON data in the payload and deserializes it to a LinkData type that we can use easily elsewhere in the code. This is the easy part. Let’s look now to the end of the function, where things start to get interesting.
Let’s skip over some code and look at the final block, because this is where things get interesting.
let! (result: Result<ID,string>) =
tweet
>>== toot
>>== sheet
return match result with
| Success s -> OkObjectResult(s) :> IActionResult
| Failure f -> StatusCodeResult(500) :> IActionResult
The >>== operator isn’t standard F# syntax. I had to define this operation. This kind of comes from the Railway Oriented Programming blog post, which introduces &&& as a parallel operator. I won’t explain this all here, I encourage you to read the post and absorb it. It’s brilliant. However, the methods presented in the blog post don’t work as-is for async programming. I had to modify them. Using a Code Review Stackexchange post as inspiration, I defined the >>== operator to do async parallel railway switches, because with the Fira Code ligatures I use, it looks nicely parallel.
To make this work at all, I have to introduce a couple helper functions and types.
type Result<'TSuccess,'TFailure> =
| Success of 'TSuccess
| Failure of 'TFailure
type ID = { id : string }
let bindAsync<'t,'s,'terr> (binder:'t -> Async<Result<'s,'terr>>) (result:Async<Result<'t,'terr>>) : Async<Result<'s,'terr>> =
async {
let! res = result
match res with
| Success s -> return! binder s
| Failure f -> return Failure f
}
let plusAsync addSuccess addFailure (switch1 : Async<Result<'s,'terr>>) (switch2 : Async<Result<'s,'terr>>) =
async {
let! res1 = switch1
let! res2 = switch2
return match (res1),(res2) with
| Success s1,Success s2 -> Success (addSuccess s1 s2)
| Failure f1,Success _ -> Failure f1
| Success _ ,Failure f2 -> Failure f2
| Failure f1,Failure f2 -> Failure (addFailure f1 f2)
}
let (>>@) twoTrackInput switchFunction =
bindAsync switchFunction twoTrackInput
let (>>==) v1 v2 =
let addSuccess r1 r2 = {id=r1.id + "; " + r2.id}
let addFailure s1 s2 = s1 + "; " + s2
plusAsync addSuccess addFailure v1 v2
The key here is the introduction of the Result<'TSuccess, 'TFailure> type and the terribly named ID record type, which will capture relevant information for successful posts; namely, the post ID.
The >>@ function is an asynchronous (hence the @) version of the >>= bind-with-piping operator, which calls the bindAsync function which, as you can guess, is the async equivalent of the bind operator from the original blog post. The plusAsync function is the asynchronous version of the plus function from the original blog post. All in all, if you can grok the Railway Oriented Programming post, these modifications are not that much more complex. They’re simply adapted for the Async<Result<'s,'terr>> return type.
When we get to the end of the code, we have:
return match result with
| Success s -> OkObjectResult(s) :> IActionResult
| Failure f -> StatusCodeResult(500) :> IActionResult
This is a really nice way to either return a 200 OK status code, along with a message that includes our successful IDs, or a 500 Server Error code, which, unfortunately, I haven’t figured out how to better manage.
If you’ve made it this far, then the remaining code in this function should be very easy to understand.
let tweet: Async<Result<ID,string>> =
getTwitterClient
>>@ writeTweet data
let sheet: Async<Result<ID,string>> =
getSheetService
>>@ writeToGoogleSheet data
let toot: Async<Result<ID,string>> =
getMastodonClient
>>@ writeToot data
All that this does is define functions that use the >>@ async bind-with-pipe operator to post the link to the various services. I really like this code. The run method is very clean, and aside from the subtleties of the syntax, the structure of the code makes it very clear what is happening. A little bit of work makes the code much, much cleaner. All we need to do now is write the code for each of these services. Ready for that? Let’s move onto the next part.
For my link sharing function, I want to push content to (at least) three different sinks: a Google Sheets, which serves as kind of a bootleg database; Twitter; and Mastodon. To do this, I’ll use the Google Sheets API, Tweetinvi, and Mastonet. These are all fairly easy to install with the dotnet package manager.
One of the beautiful things about F# is that you can use any .Net library with it, even if it wasn’t built for F#. That means any C# library works with F# equally well. This is a long post and I’m not going to belabor how to do all of these things, but rather, I’ll cover the key steps and present the code.
This is by far the most complex integration because to make this work you need to use Google Cloud. Start by creating a new Google Sheet (you can just go to sheets.new), or if you don’t have a Gmail account, create one first and then create a new Google Sheet. You can name it whatever, I called mine “Interesting Links”. In the first row, name the columns as follows: url, title, added, comment.
Next, navigate to the Google Cloud console. Create a new project, and call it something obvious, like Interesting Links. Select the project and navigate to the project dashboard. In the hamburger menu in the top left, go to “APIs & Services” and click “Enabled APIs & Services”. There, at the top of the screen, click “+ Enable APIs and Services”. In the search bar, click “Google Sheets” and enable the API.
Once done, click “Manage”. On the left, you should see “Credentials”. Click this and you should see “API Keys”, “OAuth 2.0 Client IDs”, and “Service Accounts”. We’ll use a Service Account, although this may not be the best option. Perhaps in a future update I will fix this. Click “Create Credentials” at the top and choose “Service Account”.
Give the Service Account a meaningful name, and click “Create and Continue.” On the next screen, it will ask you to choose a role. In the search box, type “Service Account Token Creator”. Skip the next step and click “Done”. This will return you to the Credentials screen. Click the Service Account you created, and at the top of the screen, click “Keys.” Next, click “Add Key” and “Create New Key”. This should pop a dialog box. Select “JSON” and click “Create”. This will create a JSON file for you and download it. Copy it somewhere meaningful, such as ~/.config/gcloud.
We’ll base64 this string, and the easiest way with the command line is simply base64 -w 0 ~/.config/gcloud/[KEY FILE].json.
For local testing, we can either create an environment variable with this file, or we can put it in local.settings.json with a field named GOOGLE_SERVICE_ACCOUNT_CREDENTIAL. While we’re doing this, go to the spreadsheet you created and copy the sheet ID. It’s the long alphanumeric string that should be between d/ and /edit. Put this in a variable called GOOGLE_SPREADSHEET_ID.
Lastly, copy the Service Account email you created, which can be found on the Credentials page we visited earlier. In your Google Sheet, in the top right click “Share” and paste this email, giving it Editor permissions.
All of this could have probably been done with terraform, but I’m going to be real. This is a one-time setup, and it’s complicated. I might have gotten it wrong here, but you can also Google how to set up a Google Sheets API credential. I don’t think it’s worth figuring out the terraform. Alternatively, we could have used Workload Identity Federation, but this, too, is overkill.
Once we have the account set up, we can write the code to write to a spreadsheet. It’s fairly straightforward. First, we have to build a SheetsService using the Google Cloud SDK. This requires our credential. Here, I define getSheetService which does all of the work to turn the environment variable into a credential and uses this to initialize a SheetsService. Here, we’re returning a Success type containing the object we just initialized, so we need to create a record type, Sheet, for that as well.
open Google.Apis.Auth.OAuth2
open Google.Apis.Sheets.v4
open Google.Apis.Sheets.v4.Data
open Google.Apis.Services
// ...
type Sheet = { service : SheetsService }
// ...
let getSheetService =
async {
let scopes: string list = [ SheetsService.Scope.Spreadsheets ]
let getServiceAccountCredential (googleCredential: GoogleCredential) =
googleCredential.CreateScoped(scopes).UnderlyingCredential
:?> ServiceAccountCredential
try
let credential : ServiceAccountCredential =
System.Environment.GetEnvironmentVariable("GOOGLE_SERVICE_ACCOUNT_CREDENTIAL")
|> Convert.FromBase64String
|> Text.Encoding.UTF8.GetString
|> GoogleCredential.FromJson
|> getServiceAccountCredential
let initializer: BaseClientService.Initializer =
new BaseClientService.Initializer(HttpClientInitializer=credential,
ApplicationName=ApplicationName)
return Success {service=new SheetsService(initializer)}
with
| ex -> return Failure ex.Message
}
let writeToGoogleSheet (data: LinkData) (input : Sheet) =
async {
let service: SheetsService = input.service
let sheetId: string = System.Environment.GetEnvironmentVariable("GOOGLE_SHEET_ID")
let range: string = "A:D"
let newItem: List<IList<Object>> = new List<IList<Object>>();
let obj: List<Object> = new List<Object>([|data.url :> Object;
data.title;
DateTime.UtcNow;
data.comment|]);
newItem.Add(obj)
let request =
service.Spreadsheets.Values.Append(new ValueRange(Values=newItem),
sheetId,
range,
InsertDataOption=SpreadsheetsResource.ValuesResource.AppendRequest.InsertDataOptionEnum.INSERTROWS,
ValueInputOption=SpreadsheetsResource.ValuesResource.AppendRequest.ValueInputOptionEnum.USERENTERED)
try
let! response = request.ExecuteAsync() |> Async.AwaitTask
return Success {id=response.Updates.UpdatedRange}
with
| (ex: exn) -> return Failure ex.Message
}
In writeGoogleSheets, we consider each row to be a list of objects, and each sheet range to be a list of rows, and hence a list of lists of objects. I know I have four columns, so I can simply define my sheet range using open-ended columns, or A:D. We create a new row with the relevant fields, url, title, added, and comment, although they appear here unnamed, and simply append this using the API. We construct the request, and by using try-with, we can either return a Success if the operation succeeds, or a Failure containing an exception message if the operation fails.
This code is again very simple. We’re simply fetching a credential, initializing a sheet service, crafting the data to be added, and adding it.
Google Sheets was very complicated, but Twitter is a bit easier. To post to Twitter, the important thing to know is you’ll need an Individual Developer Account with elevated privileges. This can be done at the Twitter Developer Portal, but I’ll be honest the portal is badly designed and things are really hard to find. It took me an hour, and now that I’ve done it once, I can’t go back to figure out how to tell you how to do it. All I can tell you are the important words: individual developer account and elevated privileges. Click around. Godspeed.
You’ll need to create a new project, I called mine “Emily’s Syndicator”. This allows me to post statuses to Twitter. Once you do this, on the left you should see your project. Click it, and toward the top of the screen you should see “Keys and tokens.” Here, you can generate the required keys. I’m using an “Access token and secret”, which I copy to a JSON file locally and also can put in local.settings.json in fields called TWITTER_ACCESS_TOKEN and TWITTER_ACCESS_TOKEN_SECRET for local testing.
Next, generate an API key and secret, sometimes called a consumer key and secret, and store these locally in fields named TWITTER_CONSUMER_KEY and TWITTER_CONSUMER_KEY_SECRET.
There are better ways to authenticate with Twitter, but everything is badly documented. This works, and I’ll improve it later. To post, I’m using the Tweetinvi SDK. Tweetinvi hasn’t been updated in a couple of years, but it works. Just like with Google Sheets, we’ll need to create a record type to receive the Success case, otherwise the workflow is even simpler.
open Tweetinvi
// ...
type Twitter = { twitterClient : TwitterClient }
// ...
let buildPostFromData (data: LinkData) =
data.comment + "\n\n" + data.url
// ...
let getTwitterClient =
async {
try
return Success {twitterClient=new TwitterClient(System.Environment.GetEnvironmentVariable("TWITTER_CONSUMER_KEY"),
System.Environment.GetEnvironmentVariable("TWITTER_CONSUMER_KEY_SECRET"),
System.Environment.GetEnvironmentVariable("TWITTER_ACCESS_TOKEN"),
System.Environment.GetEnvironmentVariable("TWITTER_ACCESS_TOKEN_SECRET"))}
with
| ex -> return Failure ex.Message
}
let writeTweet (data: LinkData) (input : Twitter) =
async {
let client = input.twitterClient
let tweet = buildPostFromData data
try
let post = client.Tweets.PublishTweetAsync(tweet)
return Success {id=post.Result.Id.ToString()}
with
| ex -> return Failure ex.Message
}
This code is extremely straightforward. First, we construct the client using the Twitter credentials obtained as described above. Then, we construct a post from the comment and url fields in the JSON payload. Last, we publish the Tweet and return the Tweet ID. It really could not be simpler.
Mastodon is even easier than Twitter. The setup is quite simple. From your Mastodon account on web, click on “Preferences” and then choose “Development”. Create a new application, give it a meaningful name, and unselect all scopes except write:statuses. Click “Submit” and you’ll retrieve an Authentication token. Store this in a local environment variable (or in local.settings.json, as before) called MASTODON_ACCESS_TOKEN and while we’re here, put your server name in as well, as MASTODON_SERVER.
Once complete, we’ll follow nearly the identical steps we did for Twitter, just using the Mastonet SDK instead.
open Mastonet
// ...
type Mastodon = { mastodonClient: MastodonClient }
// ...
let getMastodonClient =
async {
try
let client = new MastodonClient(System.Environment.GetEnvironmentVariable("MASTODON_SERVER"),
System.Environment.GetEnvironmentVariable("MASTODON_ACCESS_TOKEN"))
return Success { mastodonClient=client }
with
| ex -> return Failure ex.Message
}
let writeToot (data: LinkData) (input: Mastodon) =
async {
let client = input.mastodonClient
let post = buildPostFromData data
try
let result = client.PublishStatus(post).Result
return Success { id=result.Id }
with
| ex -> return Failure ex.Message
}
Piece of cake.
With each of these, notice how I split the process up into two steps: first we construct the client or service, and then we execute the posting. These are all asynchronous operations, and we chain them together using the >>@ async bind-with-piping operator. We bind each of those, which lets us then parallelize them as described earlier. Altogether, the code should be very easy and very minimal. The hard part is configuring the various accounts.
Once we put this all together, it should come out to ~200 lines of code. I’ve copied my file (which might have some small differences) to a public Gist.
If everything is working right, then you should be able to execute this locally fairly easily. As before, run func host start, and it should fire up a webserver at localhost:7071. Use a cURL request to post a JSON payload as follows:
curl -X POST http://localhost:7071/api/LinkSharer -d '{"url": "test", "title": "test title", "comment": "test comment"}' -v
If everything works, go check your social media services! You should see test messages. Don’t forget to delete these messages, of course ;)
This was a very long post, but hopefully you followed it. I tried to structure it step-by-step. Of course, you can skip any of the social media services. If you don’t want to use Google Sheets as a data store, then you can just ignore all that! Altogether this isn’t a lot of code, and I like how the code can be very structured and clean. There’s very little boilerplate, and you can really focus on the functional flow of the code, rather than state management and object handling. The railway pattern works really well with asynchronous programming. I like this result, even if the setup was a pain in the ass.
We’re still not out of the woods yet. We still need to set up Azure to deploy this function to a live environment. I’ll write that post tomorrow.
For now, I hope you liked this F#-based approach, and thanks for sticking it out with this long post!
Posted: 27.12.2022
Built: 21.12.2024
Updated: 24.04.2023
Hash: 765e2d9
Words: 3832
Estimated Reading Time: 20 minutes