[ This is a content summary only. Visit our website http://ift.tt/1b4YgHQ for full links, other content, and more! ]
by Web Desk via Digital Information World
Saturday, March 10, 2018
Jack Ma’s journey - #Infographic
Diminutive Jack Ma speaks softly – yet colorfully – and carries a big stick as the founder of China’s most popular online store. When he set up Alibaba, he saw a chance to connect China’s manufacturers to wholesale buyers around the world, fulfilling the ambition he formed as soon as he heard...
[ This is a content summary only. Visit our website http://ift.tt/1b4YgHQ for full links, other content, and more! ]
by Web Desk via Digital Information World
[ This is a content summary only. Visit our website http://ift.tt/1b4YgHQ for full links, other content, and more! ]
by Web Desk via Digital Information World
Friday, March 9, 2018
4 Tips for Staying Focused on What Matters When You Work From Home [video]
Many companies are allowing employees to work from home. In fact, 43 percent of American workers said they worked from home at least some of the time in the last year. And there's an entire group of workers -- among them entrepreneurs and those running startups -- who operate exclusively from...
[ This is a content summary only. Visit our website http://ift.tt/1b4YgHQ for full links, other content, and more! ]
by Web Desk via Digital Information World
[ This is a content summary only. Visit our website http://ift.tt/1b4YgHQ for full links, other content, and more! ]
by Web Desk via Digital Information World
Hey, I’m Phil
Unique One Pager for Phillip Pastore featuring tons of interactive text that overlays media. It’s got personality for sure and neat added touch stacking all the media in a hidden footer (that’s also draggable).
by Rob Hope @robhope via One Page Love
#376: This week's JavaScript news smoosh-up
|
by via JavaScript Weekly
11 Pro Email Marketing Tips - #infographic
There’s a lot to learn when you’re new to email marketing. Wouldn’t it be great to get some wisdom from an email marketing pro? Get 11 email marketing tips from some of the best in the business.
[ This is a content summary only. Visit our website http://ift.tt/1b4YgHQ for full links, other content, and more! ]
by Web Desk via Digital Information World
[ This is a content summary only. Visit our website http://ift.tt/1b4YgHQ for full links, other content, and more! ]
by Web Desk via Digital Information World
Tiffany Break
Parallax Scrolling, media-rich Landing Page for the ‘House of Crunch’ campaign featuring Tiffany Break celebrities.
by Rob Hope @robhope via One Page Love
Cleaning Up Your Data With Go: Part 1
Overview
One of the most important aspects of any application is validating its input. The most basic approach is just failing if the input doesn't satisfy the requirements. However, in many cases this is not enough. In many systems the data collection is separate from data analysis. It could be a survey or an old dataset.
In these cases, it is necessary to go over the entire dataset before analysis, detect invalid or missing data, fix what can be fixed, and flag or remove data that can't be salvaged. It is also useful to provide statistics about the quality of the data and what kinds of errors were encountered.
In this two-part series you'll learn how to use Go's text facilities, slice and dice CSV files, and ensure your data is spotlessly clean. In part one, we'll focus on the foundation of text processing in Go—bytes, runes, and strings—as well as working with CSV files.
Text in Go
Before we dive into data cleaning, let's start with the foundation of text in Go. The building blocks are bytes, runes, and strings. Let's see what each one represents and what the relationships are between them.
Bytes
Bytes are 8-bit numbers. Each byte can represent one of a possible 256 values (2 to the power of 8). Each character in the ASCII character set can be represented by a single byte. But bytes are not characters. The reason is that Go as a modern language supports Unicode, where there are way more than 256 separate characters. Enter runes.
Runes
A rune in Go is another name for the int32 type. This means that each rune can represent more than four billion separate values (2 to the power of 32), which is good enough to cover the entire Unicode character set.
In the following code you can see that the rune '∆' (alt-J on the Mac) is just an int32. To print the character it represents to the screen, I have to convert it to a string.
package main
import (
"fmt"
)
func main() {
r := '∆'
fmt.Println(r)
fmt.Println(int32(r))
fmt.Println(string(r))
}
Output:
8710
8710
∆
Unicode is complicated. A rune officially represents a Unicode code point. Unicode characters are usually represented by a single Unicode code point, but sometimes more than one.
Strings
Strings are officially just read-only slices of bytes. If you index a string, you get a byte back:
func main() {
s := "abc"
for i := 0; i < len(s); i++ {
fmt.Println(s[i])
}
}
Output:
97
98
99
String literals are a sequence of UTF-8 characters enclosed in double quotes. They may contain escape sequences, which are a backslash followed by an ASCII character such as \n (newline) or \t (tab). They have special meanings. Here is the full list:
\a U+0007 alert or bell \b U+0008 backspace \f U+000C form feed \n U+000A line feed or newline \r U+000D carriage return \t U+0009 horizontal tab \v U+000b vertical tab \\ U+005c backslash \' U+0027 single quote (valid only within rune literals) \" U+0022 double quote (valid only within string literals)
Sometimes you may want to store literal bytes directly in a string, regardless of escape sequences. You could escape each backslash, but that's tedious. A much better approach is to use raw strings that are enclosed in backticks.
Here is an example of a string with a \t (tab) escape sequence, which is represented once as is, then with the backslash escape, and then as a raw string:
func main() {
s1 := "1\t2"
s2 := "1\\t2"
s3 := `1\t2`
fmt.Println(s1)
fmt.Println(s2)
fmt.Println(s3)
}
Output:
1 2
1\t2
1\t2
While strings are slices of bytes, when you iterate over a string with a for-range statement, you get a rune in each iteration. This means you may get one or more bytes. This is easy to see with the for-range index. Here is a crazy example. The Hebrew word "שלום" means "Hello" (and peace). Hebrew is also written right to left. I'll construct a string that mixes the Hebrew word with its English translation.
Then, I'll print it rune by rune, including the byte index of each rune within the string. As you'll see, each Hebrew rune takes two bytes, while the English characters take one byte, so the total length of this string is 16 bytes, even though it has four Hebrew characters, three symbols, and five English characters (12 characters). Also, the Hebrew characters will be displayed from right to left:
func main() {
hello := "שלום = hello"
fmt.Println("length:", len(hello))
for i, r := range(hello) {
fmt.Println(i, string(r))
}
}
Output:
length: 16
0 ש
2 ל
4 ו
6 ם
8
9 =
10
11 h
12 e
13 l
14 l
15 o
All these nuances can be extremely important when you have a dataset to clean up with weird quotes and a mix of Unicode characters and symbols.
When printing strings and byte slices, there are several format specifiers that work the same on both. The %s format prints the bytes as is, %x prints two lowercase hexadecimal characters per byte, %X prints two uppercase hexadecimal characters per byte, and %q prints a double quoted string escaped with go syntax.
To escape the % sign inside a format string specifier, just double it. To separate the bytes when using %x or %X, you can add a space, as in "% x" and "% X". Here is the demo:
func main() {
s := "שלום"
fmt.Printf("%%s format: %s\n", s)
fmt.Printf("%%x format: %x\n", s)
fmt.Printf("%%X format: %X\n", s)
fmt.Printf("%% x format: % x\n", s)
fmt.Printf("%% X format: % X\n", s)
fmt.Printf("%%q format: %q\n", s)
}
Output:
%s format: שלום
%x format: d7a9d79cd795d79d
%X format: D7A9D79CD795D79D
% x format: d7 a9 d7 9c d7 95 d7 9d
% X format: D7 A9 D7 9C D7 95 D7 9D
%q format: "שלום"
Reading and Writing CSV Files
Data can arrive in many ways and formats. One of the most common formats is CSV (comma-separated values). CSV data is very efficient. The files typically have a header line with the name of the fields or columns and rows of data where each row contains a value per field, separated by commas.
Here is a little snippet from a UFO sightings dataset (really). The first row (header) contains the column names, and the other lines contain the data. You can see that often the "Colors Reported" column is empty:
City,Colors Reported,Shape Reported,State,Time Ithaca,,TRIANGLE,NY,6/1/1930 22:00 Willingboro,,OTHER,NJ,6/30/1930 20:00 Holyoke,,OVAL,CO,2/15/1931 14:00 Abilene,,DISK,KS,6/1/1931 13:00 New York Worlds Fair,,LIGHT,NY,4/18/1933 19:00 Valley City,,DISK,ND,9/15/1934 15:30 Crater Lake,,CIRCLE,CA,6/15/1935 0:00 Alma,,DISK,MI,7/15/1936 0:00 Eklutna,,CIGAR,AK,10/15/1936 17:00 Hubbard,,CYLINDER,OR,6/15/1937 0:00 Fontana,,LIGHT,CA,8/15/1937 21:00 Waterloo,,FIREBALL,AL,6/1/1939 20:00 Belton,RED,SPHERE,SC,6/30/1939 20:00
Writing this chunk of CSV data to a file involves some string operations as well as working with files. Before we dive into the main logic, here are the mandatory parts: the package definition, the imports, and the data string (note the use of const).
package main
import (
"os"
"strings"
"bufio"
)
data := `
City,Colors Reported,Shape Reported,State,Time
Ithaca,,TRIANGLE,NY,6/1/1930 22:00
Willingboro,,OTHER,NJ,6/30/1930 20:00
Holyoke,,OVAL,CO,2/15/1931 14:00
Abilene,,DISK,KS,6/1/1931 13:00
New York Worlds Fair,,LIGHT,NY,4/18/1933 19:00
Valley City,,DISK,ND,9/15/1934 15:30
Crater Lake,,CIRCLE,CA,6/15/1935 0:00
Alma,,DISK,MI,7/15/1936 0:00
Eklutna,,CIGAR,AK,10/15/1936 17:00
Hubbard,,CYLINDER,OR,6/15/1937 0:00
Fontana,,LIGHT,CA,8/15/1937 21:00
Waterloo,,FIREBALL,AL,6/1/1939 20:00
Belton,RED,SPHERE,SC,6/30/1939 20:00
`
The main() function creates a file called "ufo-sightings.csv", checks that there is no error, and then creates a buffered writer w. The defer call in the next line, which flushes the contents of the buffer to the file, is executed at the end of the function. That is the meaning of defer. Then, it uses the Split() function of the strings package to break the data strings into individual lines.
Then, inside the for-loop, the leading and trailing whitespace is trimmed from each line. Empty lines are skipped, and non-empty lines are written to the buffer, followed by a newline character. That's it. The buffer will be flushed to the file in the end.
func main() {
f, err := os.Create("ufo-sightings.csv")
if err != nil {
panic(e)
}
w := bufio.NewWriter(f)
defer w.Flush()
lines := strings.Split(data, "\n")
for _, line := range lines {
line := strings.Trim(line, " ")
if line == "" {
continue
}
w.WriteString(line)
w.WriteString("\n")
}
}
Reading from the file is pretty simple:
package main
import (
"fmt"
"io/ioutil"
)
func main() {
data, err := ioutil.ReadFile("ufo-sightings.csv")
if err != nil {
panic(err)
}
fmt.Println(string(data))
}
Conclusion
Go has strong facilities to deal with text of all shapes and encodings. In this part of the series, we looked into the basics of text representation in Go, text processing using the strings package, and dealing with CSV files.
In part two, we will put what we've learned into practice to clean up messy data in preparation for analysis.
by Gigi Sayfan via Envato Tuts+ Code
Subscribe to:
Comments (Atom)