//blog.sgmansfield.com/ Recent content on Scott Mansfield Hugo -- gohugo.io en-us Fri, 28 Apr 2017 00:00:00 -0700 //blog.sgmansfield.com/2017/04/nanolog-super-fast-logging-for-go/ Fri, 28 Apr 2017 00:00:00 -0700 //blog.sgmansfield.com/2017/04/nanolog-super-fast-logging-for-go/ I’ve released a new package called nanolog! Nanolog is super fast, as low as 70ns to log a line, and works with concurrent loggers. It uses a somewhat different model than most people are used to for the API; you can think of it like prepared statements for logging. It is currently at version 0.1.0. This package was inspired by a project of the same name for C++. Why? Because. //blog.sgmansfield.com/2017/04/a-foray-into-go-assembly-programming/ Fri, 21 Apr 2017 03:37:23 -0700 //blog.sgmansfield.com/2017/04/a-foray-into-go-assembly-programming/ This blog post started last August when I was integrating the Spectator PercentileTimer concept into the metrics library in Rend so we could get better cross-fleet latency percentiles. As a part of doing this, I had to port the code that selects which bucket (counter) to increment inside the PercentileTimer distribution. A PercentileTimer is implemented as a large array of counters, each of which represent a bucket. They are incremented whenever the observation lands in that bucket. //blog.sgmansfield.com/2016/12/goroutree-a-tree-based-set-made-of-coordinating-goroutines/ Wed, 14 Dec 2016 00:00:00 +0000 //blog.sgmansfield.com/2016/12/goroutree-a-tree-based-set-made-of-coordinating-goroutines/ This is a cross-post of my blog post for the GopherAcademy Advent 2016 series. It’s replicated below in its entirety. This was one of those projects that sat in the back of my mind for quite a while. It was destined to join the many others in my side project graveyard unless I had a good reason to finish it, like a date for a blog post. This post is an explanation and exploration of the goroutree data structure. //blog.sgmansfield.com/2016/08/10k-concurrent-connections/ Fri, 05 Aug 2016 01:09:22 -0700 //blog.sgmansfield.com/2016/08/10k-concurrent-connections/ After my recent appearance on the Go Time podcast I thought it would be fun to write a post on a few performance bits in Rend, the software project I hack on at work. This topic comes up frequently enough that I felt I should write about it. The c10k problem has many solutions already. This is just my take. Fortunately for people in 2016, handling 10 thousand concurrent connections is not really all that difficult anymore. //blog.sgmansfield.com/2016/06/working-with-forks-in-go/ Sat, 18 Jun 2016 21:17:27 -0700 //blog.sgmansfield.com/2016/06/working-with-forks-in-go/ This been written about before in the Go universe, but I felt it was worth reviewing as it has come up many times in the Go slack (invite link). When developing on a separate remote from the original in Go, there is a common point of confusion when a developer forks on Github (for example) and then proceeds to go get the fork. This places the fork in the wrong spot in their $GOPATH and none of the imports work. //blog.sgmansfield.com/2016/06/managing-syscall-overhead-with-crypto-rand/ Fri, 10 Jun 2016 00:12:35 -0700 //blog.sgmansfield.com/2016/06/managing-syscall-overhead-with-crypto-rand/ The overhead of using secure random numbers can be a headache if the generation of those numbers is in your server’s critical path. In this post, I’ll look at a couple of techniques to bypass the overhead of generating random numbers in a Go program and make a recommendation on what method to use. Consider an application that needs to generate a nonce for each request which is also I/O bound, meaning it does more waiting on I/O than anything else. //blog.sgmansfield.com/2016/06/how-to-block-forever-in-go/ Sun, 05 Jun 2016 00:17:35 -0700 //blog.sgmansfield.com/2016/06/how-to-block-forever-in-go/ Let me count the ways There seems to be quite a few ways to block forever in Go. This post is part code golf and part practical advice. All of the code below is simply to find a way to block the current goroutine from progressing while still allowing others to continue. The versions that block forever will actually cause a panic saying that all goroutines are asleep, but the ones that “busy block” will just time out on the playground. //blog.sgmansfield.com/2016/01/locking-in-crypto-rand/ Sun, 10 Jan 2016 19:55:30 -0800 //blog.sgmansfield.com/2016/01/locking-in-crypto-rand/ As a followup to my previous post detailing my journey through some profiling of the math/rand package, I wanted to write about the crypto/rand package. A couple people have suggested that I take a look at that instead of worrying about locking in math/rand. On the surface, it’s an easy interface that fills a byte slice full of cryptographically secure random data. I modified the rand_default.go program from the previous post to create a new program to pull data from crypto/rand instead of math/rand. //blog.sgmansfield.com/2016/01/the-hidden-dangers-of-default-rand/ Tue, 05 Jan 2016 01:15:59 -0800 //blog.sgmansfield.com/2016/01/the-hidden-dangers-of-default-rand/ This post is based on a real-world problem that I faced when I was developing a load generator program for work. I looked through my code carefully to make sure that there weren’t any bottlenecks in the design or bugs in the implementation, but didn’t find anything explicitly wrong. I did some profiling after that and found something very interesting: deep down in the rand package was a lock that was taking up the majority of the time in the application. //blog.sgmansfield.com/2016/01/an-analysis-of-the-top-1000-go-repositories/ Sat, 02 Jan 2016 01:25:41 -0800 //blog.sgmansfield.com/2016/01/an-analysis-of-the-top-1000-go-repositories/ This analysis was done from copies cloned on January 2, 2016 early morning Pacific Time. Code organization Most code is a library, so the code is organized as either .go files under the main repo, or as .go files under sub-directories. Many people also organize their code under a sub-directory, like /src, /lib/, /go/, or /pkg/. I can’t manually inspect all of the repositories, but those I did check are apps written in go rather than libraries. //blog.sgmansfield.com/2015/12/the-insanity-of-generating-all-possible-guids/ Sat, 26 Dec 2015 08:46:53 -0800 //blog.sgmansfield.com/2015/12/the-insanity-of-generating-all-possible-guids/ Genesis There was once a StackOverflow question (“was” because it’s gone now) title “Fastest way in C# to iterate through ALL Guids Possible”. The premise was that the person wanted to hit a web server with a single request per GUID to determine if the server had data for it. Hilarity ensued. Suggestions Some people were legitimately trying to be helpful in this quixotic quest. Some pointed out: Floating point math mistakes: //blog.sgmansfield.com/2015/12/widow-web-crawler-architecture/ Fri, 18 Dec 2015 00:09:52 -0800 //blog.sgmansfield.com/2015/12/widow-web-crawler-architecture/ Widow In a previous post, I went over the justification for building my own web crawler named Widow. Here I will explain my alternative method for building a large-scale web crawler. Keep in mind that the crawler is still a work in progress (as of the end of 2015) so this is not final. There is still some future work to be done, which will be discussed at the end. //blog.sgmansfield.com/2015/12/goroutine-ids/ Wed, 16 Dec 2015 01:50:55 -0800 //blog.sgmansfield.com/2015/12/goroutine-ids/ Do Goroutine IDs Exist? Yes, of course. The runtime has to have some way to track them. Should I use them? No. No. No. No. Are There Packages I Can Use? There exist packages from people on the Go team, with colorful descriptions like “If you use this package, you will go straight to hell.“ There are also packages to do goroutine local storage built on top of goroutine IDs, like github. //blog.sgmansfield.com/2015/12/why-i-decided-to-make-my-own-web-crawler/ Fri, 11 Dec 2015 00:39:54 -0800 //blog.sgmansfield.com/2015/12/why-i-decided-to-make-my-own-web-crawler/ Widow The web crawler I am making is named Widow, and is freely available on GitHub. Along with Widow, there are a couple of other sub-projects that were, in my mind, necessary to have a decent crawler. It must also be able to parse the robots.txt and sitemap.xml files that websites use to direct web crawler behavior. The projects to perform both of those functions are terminator and exo. They are also available on Maven Central under the com. //blog.sgmansfield.com/2012/03/automated-website-backup-to-amazon-s3/ Sat, 24 Mar 2012 00:12:29 -0500 //blog.sgmansfield.com/2012/03/automated-website-backup-to-amazon-s3/ After reading Lifehacker’s article linking to Gina Trapani’s article about automatic website backups, I decided it would be a good idea to implement this for my own websites. Gina’s solution is great for one website, but I have multiple websites under one user. I am definitely not a bash-fu master by any stretch of the imagination so the best I could have done with bash would have been to copy Gina’s script and modify it some to fit my needs. //blog.sgmansfield.com/2011/10/function-calls-word-alignment-and-interrupts-on-a-ti-dsp/ Thu, 13 Oct 2011 16:23:00 -0500 //blog.sgmansfield.com/2011/10/function-calls-word-alignment-and-interrupts-on-a-ti-dsp/ Background I was taking the microprocessors class last fall and started writing this post… now here it is! I had a couple of labs that were giving me problems. The particular DSP I was using for my lab (the Texas Instruments TMS320F28335) has a word size of 16 bits. When writing a ton of assembly code for a couple of the labs, I came across an error that cropped up concerning word alignment that was particularly obvious when using an interrupt routine set to go off every millisecond or so. //blog.sgmansfield.com/2011/09/windows-8-developer-preview-vmware-install/ Wed, 14 Sep 2011 19:00:00 -0500 //blog.sgmansfield.com/2011/09/windows-8-developer-preview-vmware-install/ I was very excited when I heard that a Developer Preview of Windows 8 was coming out. I downloaded it (even though I went over my data cap for the month) so I could play with it in a virtual machine. I immediately went to VMWare Workstation 7.14 to attempt installing it, only to run into an error that said: MONITOR PANIC: vcpu-0:NOT_IMPLEMENTED vmcore/vmm/intr/apic.c:1903 Well, crud. That didn’t work very well. //blog.sgmansfield.com/2010/09/strange-errors-makefiles-and-large-matrices/ Wed, 29 Sep 2010 15:53:00 -0500 //blog.sgmansfield.com/2010/09/strange-errors-makefiles-and-large-matrices/ Strange Errors Abound The first problems I encountered numbered in the hundreds. I was getting errors that all said something like this: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before cut… After some frantic Google searching (yes, it worked this time) I came across a forum thread that contained some lists of error messages that seemed similar to mine. Someone in this thread suggested that the compiler was compiling the CUDA code as C code so the standard library of CUDA functions was not automatically being referenced. //blog.sgmansfield.com/2010/09/cuda-timing-woes/ Sat, 25 Sep 2010 22:03:00 -0500 //blog.sgmansfield.com/2010/09/cuda-timing-woes/ I have come to realize that programming using Nvidia’s CUDA can be a major PITA pit when a problem crops up. Generally there are few resources besides the CUDA Programming Guide that are helpful. Most of the answers that one would find in their searches are on the forums at Nvidia, and even then many searches turn up unanswered questions. My particular problem was in the realm of timing the execution of a kernel on the device.