Twitter incremental backup in YAML format - python, HTML get and parse

I use Twitter primarly as a bookmarking service.

I know that Twitter is not bookmarking service and there are specialized services for such needs. In my case, using this way started as an experiment, but later I found this way very convenient.

I wrote a script that enables me to incrementally backup all my tweets in a textual file in YAML format, by parsing HTML Twitter pages. Explanation and the script follows …

My case

So this is my case:

  • I read a lot
  • Google Reader
  • HTC Desire smartphone
  • everything I found interesting is sent to Twitter by using Android Peep Twitter application
  • tweets are tagged by prefixing keywords with hashes (#)
  • tweets are backuped to textual file which is held in my Dropbox folder
  • backup should be incremental
  • backup should be readable by human and machine - YAML is chosen
  • (future) all tweets/bookmarks will be publicly available on my site with advanced browsing feature
The script

Based on the script made by Scott Carpenter I made a python script that doesn’t use Twitter API, but reads Twitter HTML pages, parses them and collect tweets. On the other hand backup file is in YAML file, it is parsed and then only collected tweets that don’t exist in YAML file are written to the top of the file. The script needs BeautifulSoup HTML parsing package, and pyyaml so if you don’t have it yet:

pip install beautifulsoup
pip install pyyaml

Example of usage

Example of usage:

> dbox twitter-bak trebor74hr twitter-trebor74hr-backup.yaml
Reading http://twitter.com/trebor74hr, backup in twitter-trebor74hr-backup.yaml
started at 2011-09-24 18:30:13.937000
  1. page - read and parse
      20/ 20 tweets saved/processed. Waiting 2 seconds before fetching next page...
  2. page - read and parse
      40/ 40 tweets saved/processed. Waiting 2 seconds before fetching next page...
  3. page - read and parse
      60/ 60 tweets saved/processed. Waiting 2 seconds before fetching next page...
  4. page - read and parse
      80/ 80 tweets saved/processed. Waiting 2 seconds before fetching next page...
  5. page - read and parse
      89/100 tweets saved/processed. Waiting 2 seconds before fetching next page...
  6. page - read and parse
      No new tweets found, quit iteration ...
89/120 tweets saved in twitter-trebor74hr-backup.yaml

Try again, no new tweets:

> dbox twitter-bak trebor74hr twitter-trebor74hr-backup.yaml
Reading http://twitter.com/trebor74hr, backup in twitter-trebor74hr-backup.yaml,
started at 2011-09-24 18:30:49.906000
  1. page - read and parse
     No new tweets found, quit iteration ...
No new tweets found in 20 tweets analysed, file twitter-trebor74hr-backup.yaml

The result

The result is in twitter-trebor74hr-backup.yaml:

- content : "RT @williamtincup RT Don’t Be 'That Guy' as a Manager
            bit.ly/qJL8nx [http://t.co/fxy2IrIe] @greatleadership"
  date_time: "Fri Sep 23 20:17:32 +0000 2011"
  status_id: "117331787797102592"
  url: "http://twitter.com/#!/trebor74hr/status/117331787797102592"

- content : "python.mirocommunity.org [http://t.co/WYu9ygAb] #Python
            #Miro  Community - All Python #Video , All the Time"
  date_time: "Fri Sep 23 20:12:34 +0000 2011"
  status_id: "117330534635540481"
  url: "http://twitter.com/#!/trebor74hr/status/117330534635540481"

...


Again, the script is …

The script can be found here.

Validating YAML-style Tag Strings with Javascript

This has an admittedly very specific use-case, but recently I needed to validate strings matching the following formats:

[apples, oranges]
[[apples, oranges]]
[apples, oranges, [peaches, grapes]]

The bracket nesting is important as it will later be parsed by a YAML parser to generate nested arrays of strings

[
  "apples",
  "oranges",
  [
    "peaches",
    "grapes"
  ]
]

and then converted to SQL conditions

... WHERE tag1.tag ILIKE "apples" OR tag2.tag ILIKE "oranges" OR (tag3.tag ILIKE "peaches" AND tag4.tag ILIKE "grapes")

The user needed to be prompted as they were entering the strings in this unique format to let them know whether or not their string was valid.

validateTagFormat = function(string){
  var b = 0;
  for (i=0; i<string.length; i++) {
    var c = string[i];
    switch (c) {
      case '[':
        b++;
        break;
      case ']':
        b--;  
        break; 
    }
  }

  if (b != 0) return false;
  if (string.indexOf('[') > string.indexOf(',') && string.indexOf(',') != -1) return false;
  if (string.indexOf(',') > -1 && string.indexOf('[') == -1) return false;
  if (/\]\s*[^\],]/.test(string)) return false;

  string = string.replace(/[\[\]]/g, '');
  string = string.replace(/,\s*/g, ', ');

  return /^[a-zA-Z0\-\ ]+((\,\ )[a-zA-Z0\-\ ]+)*$/.test(string);   
};

This jsFiddle instance shows this in action.

Liste de pays FR/EN pour Doctrine

Le schema :


Country:
  actAs:
    I18n:
      fields: [title]
  columns:
    code: string(2)
    title: string(50)

et les fixtures :


Country:
  Country_1:
    code: AD
    Translation:
      fr:
        title: "Andorre"
      en:
        title: "Andorra"
  Country_2:
    code: AE
    Translation:
      fr:
        title: "Émirats arabes unis"
      en:
        title: "United Arab Emirates"
  Country_3:
    code: AF
    Translation:
      fr:
        title: "Afghanistan"
      en:
        title: "Afghanistan"
  Country_4:
    code: AG
    Translation:
      fr:
        title: "Antigua-et-Barbuda"
      en:
        title: "Antigua and Barbuda"
  Country_5:
    code: AI
    Translation:
      fr:
        title: "Anguilla"
      en:
        title: "Anguilla"
  Country_6:
    code: AL
    Translation:
      fr:
        title: "Albanie"
      en:
        title: "Albania"
  Country_7:
    code: AM
    Translation:
      fr:
        title: "Arménie"
      en:
        title: "Armenia"
  Country_8:
    code: AN
    Translation:
      fr:
        title: "Antilles néerlandaises"
      en:
        title: "Netherlands Antilles"
  Country_9:
    code: AO
    Translation:
      fr:
        title: "Angola"
      en:
        title: "Angola"
  Country_10:
    code: AQ
    Translation:
      fr:
        title: "Antarctique"
      en:
        title: "Antarctica"
  Country_11:
    code: AR
    Translation:
      fr:
        title: "Argentine"
      en:
        title: "Argentina"
  Country_12:
    code: AS
    Translation:
      fr:
        title: "Samoa américaines"
      en:
        title: "American Samoa"
  Country_13:
    code: AT
    Translation:
      fr:
        title: "Autriche"
      en:
        title: "Austria"
  Country_14:
    code: AU
    Translation:
      fr:
        title: "Australie"
      en:
        title: "Australia"
  Country_15:
    code: AW
    Translation:
      fr:
        title: "Aruba"
      en:
        title: "Aruba"
  Country_16:
    code: AX
    Translation:
      fr:
        title: "Îles Åland"
      en:
        title: "Åland Islands"
  Country_17:
    code: AZ
    Translation:
      fr:
        title: "Azerbaïdjan"
      en:
        title: "Azerbaijan"
  Country_18:
    code: BA
    Translation:
      fr:
        title: "Bosnie-Herzégovine"
      en:
        title: "Bosnia and Herzegovina"
  Country_19:
    code: BB
    Translation:
      fr:
        title: "Barbade"
      en:
        title: "Barbados"
  Country_20:
    code: BD
    Translation:
      fr:
        title: "Bangladesh"
      en:
        title: "Bangladesh"
  Country_21:
    code: BE
    Translation:
      fr:
        title: "Belgique"
      en:
        title: "Belgium"
  Country_22:
    code: BF
    Translation:
      fr:
        title: "Burkina Faso"
      en:
        title: "Burkina Faso"
  Country_23:
    code: BG
    Translation:
      fr:
        title: "Bulgarie"
      en:
        title: "Bulgaria"
  Country_24:
    code: BH
    Translation:
      fr:
        title: "Bahreïn"
      en:
        title: "Bahrain"
  Country_25:
    code: BI
    Translation:
      fr:
        title: "Burundi"
      en:
        title: "Burundi"
  Country_26:
    code: BJ
    Translation:
      fr:
        title: "Bénin"
      en:
        title: "Benin"
  Country_27:
    code: BL
    Translation:
      fr:
        title: "Saint-Barthélémy"
      en:
        title: "Saint Barthélemy"
  Country_28:
    code: BM
    Translation:
      fr:
        title: "Bermudes"
      en:
        title: "Bermuda"
  Country_29:
    code: BN
    Translation:
      fr:
        title: "Brunéi Darussalam"
      en:
        title: "Brunei"
  Country_30:
    code: BO
    Translation:
      fr:
        title: "Bolivie"
      en:
        title: "Bolivia"
  Country_31:
    code: BR
    Translation:
      fr:
        title: "Brésil"
      en:
        title: "Brazil"
  Country_32:
    code: BS
    Translation:
      fr:
        title: "Bahamas"
      en:
        title: "Bahamas"
  Country_33:
    code: BT
    Translation:
      fr:
        title: "Bhoutan"
      en:
        title: "Bhutan"
  Country_34:
    code: BV
    Translation:
      fr:
        title: "Île Bouvet"
      en:
        title: "Bouvet Island"
  Country_35:
    code: BW
    Translation:
      fr:
        title: "Botswana"
      en:
        title: "Botswana"
  Country_36:
    code: BY
    Translation:
      fr:
        title: "Bélarus"
      en:
        title: "Belarus"
  Country_37:
    code: BZ
    Translation:
      fr:
        title: "Belize"
      en:
        title: "Belize"
  Country_38:
    code: CA
    Translation:
      fr:
        title: "Canada"
      en:
        title: "Canada"
  Country_39:
    code: CC
    Translation:
      fr:
        title: "Îles Cocos - Keeling"
      en:
        title: "Cocos [Keeling] Islands"
  Country_40:
    code: CD
    Translation:
      fr:
        title: "République démocratique du Congo"
      en:
        title: "Congo - Kinshasa"
  Country_41:
    code: CF
    Translation:
      fr:
        title: "République centrafricaine"
      en:
        title: "Central African Republic"
  Country_42:
    code: CG
    Translation:
      fr:
        title: "Congo"
      en:
        title: "Congo - Brazzaville"
  Country_43:
    code: CH
    Translation:
      fr:
        title: "Suisse"
      en:
        title: "Switzerland"
  Country_44:
    code: CI
    Translation:
      fr:
        title: "Côte d’Ivoire"
      en:
        title: "Côte d’Ivoire"
  Country_45:
    code: CK
    Translation:
      fr:
        title: "Îles Cook"
      en:
        title: "Cook Islands"
  Country_46:
    code: CL
    Translation:
      fr:
        title: "Chili"
      en:
        title: "Chile"
  Country_47:
    code: CM
    Translation:
      fr:
        title: "Cameroun"
      en:
        title: "Cameroon"
  Country_48:
    code: CN
    Translation:
      fr:
        title: "Chine"
      en:
        title: "China"
  Country_49:
    code: CO
    Translation:
      fr:
        title: "Colombie"
      en:
        title: "Colombia"
  Country_50:
    code: CR
    Translation:
      fr:
        title: "Costa Rica"
      en:
        title: "Costa Rica"
  Country_51:
    code: CS
    Translation:
      fr:
        title: "Serbie-et-Monténégro"
      en:
        title: "Serbia and Montenegro"
  Country_52:
    code: CU
    Translation:
      fr:
        title: "Cuba"
      en:
        title: "Cuba"
  Country_53:
    code: CV
    Translation:
      fr:
        title: "Cap-Vert"
      en:
        title: "Cape Verde"
  Country_54:
    code: CX
    Translation:
      fr:
        title: "Île Christmas"
      en:
        title: "Christmas Island"
  Country_55:
    code: CY
    Translation:
      fr:
        title: "Chypre"
      en:
        title: "Cyprus"
  Country_56:
    code: CZ
    Translation:
      fr:
        title: "République tchèque"
      en:
        title: "Czech Republic"
  Country_57:
    code: DE
    Translation:
      fr:
        title: "Allemagne"
      en:
        title: "Germany"
  Country_58:
    code: DJ
    Translation:
      fr:
        title: "Djibouti"
      en:
        title: "Djibouti"
  Country_59:
    code: DK
    Translation:
      fr:
        title: "Danemark"
      en:
        title: "Denmark"
  Country_60:
    code: DM
    Translation:
      fr:
        title: "Dominique"
      en:
        title: "Dominica"
  Country_61:
    code: DO
    Translation:
      fr:
        title: "République dominicaine"
      en:
        title: "Dominican Republic"
  Country_62:
    code: DZ
    Translation:
      fr:
        title: "Algérie"
      en:
        title: "Algeria"
  Country_63:
    code: EC
    Translation:
      fr:
        title: "Équateur"
      en:
        title: "Ecuador"
  Country_64:
    code: EE
    Translation:
      fr:
        title: "Estonie"
      en:
        title: "Estonia"
  Country_65:
    code: EG
    Translation:
      fr:
        title: "Égypte"
      en:
        title: "Egypt"
  Country_66:
    code: EH
    Translation:
      fr:
        title: "Sahara occidental"
      en:
        title: "Western Sahara"
  Country_67:
    code: ER
    Translation:
      fr:
        title: "Érythrée"
      en:
        title: "Eritrea"
  Country_68:
    code: ES
    Translation:
      fr:
        title: "Espagne"
      en:
        title: "Spain"
  Country_69:
    code: ET
    Translation:
      fr:
        title: "Éthiopie"
      en:
        title: "Ethiopia"
  Country_70:
    code: FI
    Translation:
      fr:
        title: "Finlande"
      en:
        title: "Finland"
  Country_71:
    code: FJ
    Translation:
      fr:
        title: "Fidji"
      en:
        title: "Fiji"
  Country_72:
    code: FK
    Translation:
      fr:
        title: "Îles Malouines"
      en:
        title: "Falkland Islands"
  Country_73:
    code: FM
    Translation:
      fr:
        title: "États fédérés de Micronésie"
      en:
        title: "Micronesia"
  Country_74:
    code: FO
    Translation:
      fr:
        title: "Îles Féroé"
      en:
        title: "Faroe Islands"
  Country_75:
    code: FR
    Translation:
      fr:
        title: "France"
      en:
        title: "France"
  Country_76:
    code: GA
    Translation:
      fr:
        title: "Gabon"
      en:
        title: "Gabon"
  Country_77:
    code: GB
    Translation:
      fr:
        title: "Royaume-Uni"
      en:
        title: "United Kingdom"
  Country_78:
    code: GD
    Translation:
      fr:
        title: "Grenade"
      en:
        title: "Grenada"
  Country_79:
    code: GE
    Translation:
      fr:
        title: "Géorgie"
      en:
        title: "Georgia"
  Country_80:
    code: GF
    Translation:
      fr:
        title: "Guyane française"
      en:
        title: "French Guiana"
  Country_81:
    code: GG
    Translation:
      fr:
        title: "Guernesey"
      en:
        title: "Guernsey"
  Country_82:
    code: GH
    Translation:
      fr:
        title: "Ghana"
      en:
        title: "Ghana"
  Country_83:
    code: GI
    Translation:
      fr:
        title: "Gibraltar"
      en:
        title: "Gibraltar"
  Country_84:
    code: GL
    Translation:
      fr:
        title: "Groenland"
      en:
        title: "Greenland"
  Country_85:
    code: GM
    Translation:
      fr:
        title: "Gambie"
      en:
        title: "Gambia"
  Country_86:
    code: GN
    Translation:
      fr:
        title: "Guinée"
      en:
        title: "Guinea"
  Country_87:
    code: GP
    Translation:
      fr:
        title: "Guadeloupe"
      en:
        title: "Guadeloupe"
  Country_88:
    code: GQ
    Translation:
      fr:
        title: "Guinée équatoriale"
      en:
        title: "Equatorial Guinea"
  Country_89:
    code: GR
    Translation:
      fr:
        title: "Grèce"
      en:
        title: "Greece"
  Country_90:
    code: GS
    Translation:
      fr:
        title: "Géorgie du Sud et les îles Sandwich du Sud"
      en:
        title: "South Georgia and the South Sandwich Islands"
  Country_91:
    code: GT
    Translation:
      fr:
        title: "Guatemala"
      en:
        title: "Guatemala"
  Country_92:
    code: GU
    Translation:
      fr:
        title: "Guam"
      en:
        title: "Guam"
  Country_93:
    code: GW
    Translation:
      fr:
        title: "Guinée-Bissau"
      en:
        title: "Guinea-Bissau"
  Country_94:
    code: GY
    Translation:
      fr:
        title: "Guyana"
      en:
        title: "Guyana"
  Country_95:
    code: HK
    Translation:
      fr:
        title: "R.A.S. chinoise de Hong Kong"
      en:
        title: "Hong Kong SAR China"
  Country_96:
    code: HM
    Translation:
      fr:
        title: "Îles Heard et MacDonald"
      en:
        title: "Heard Island and McDonald Islands"
  Country_97:
    code: HN
    Translation:
      fr:
        title: "Honduras"
      en:
        title: "Honduras"
  Country_98:
    code: HR
    Translation:
      fr:
        title: "Croatie"
      en:
        title: "Croatia"
  Country_99:
    code: HT
    Translation:
      fr:
        title: "Haïti"
      en:
        title: "Haiti"
  Country_100:
    code: HU
    Translation:
      fr:
        title: "Hongrie"
      en:
        title: "Hungary"
  Country_101:
    code: ID
    Translation:
      fr:
        title: "Indonésie"
      en:
        title: "Indonesia"
  Country_102:
    code: IE
    Translation:
      fr:
        title: "Irlande"
      en:
        title: "Ireland"
  Country_103:
    code: IL
    Translation:
      fr:
        title: "Israël"
      en:
        title: "Israel"
  Country_104:
    code: IM
    Translation:
      fr:
        title: "Île de Man"
      en:
        title: "Isle of Man"
  Country_105:
    code: IN
    Translation:
      fr:
        title: "Inde"
      en:
        title: "India"
  Country_106:
    code: IO
    Translation:
      fr:
        title: "Territoire britannique de l'océan Indien"
      en:
        title: "British Indian Ocean Territory"
  Country_107:
    code: IQ
    Translation:
      fr:
        title: "Irak"
      en:
        title: "Iraq"
  Country_108:
    code: IR
    Translation:
      fr:
        title: "Iran"
      en:
        title: "Iran"
  Country_109:
    code: IS
    Translation:
      fr:
        title: "Islande"
      en:
        title: "Iceland"
  Country_110:
    code: IT
    Translation:
      fr:
        title: "Italie"
      en:
        title: "Italy"
  Country_111:
    code: JE
    Translation:
      fr:
        title: "Jersey"
      en:
        title: "Jersey"
  Country_112:
    code: JM
    Translation:
      fr:
        title: "Jamaïque"
      en:
        title: "Jamaica"
  Country_113:
    code: JO
    Translation:
      fr:
        title: "Jordanie"
      en:
        title: "Jordan"
  Country_114:
    code: JP
    Translation:
      fr:
        title: "Japon"
      en:
        title: "Japan"
  Country_115:
    code: KE
    Translation:
      fr:
        title: "Kenya"
      en:
        title: "Kenya"
  Country_116:
    code: KG
    Translation:
      fr:
        title: "Kirghizistan"
      en:
        title: "Kyrgyzstan"
  Country_117:
    code: KH
    Translation:
      fr:
        title: "Cambodge"
      en:
        title: "Cambodia"
  Country_118:
    code: KI
    Translation:
      fr:
        title: "Kiribati"
      en:
        title: "Kiribati"
  Country_119:
    code: KM
    Translation:
      fr:
        title: "Comores"
      en:
        title: "Comoros"
  Country_120:
    code: KN
    Translation:
      fr:
        title: "Saint-Kitts-et-Nevis"
      en:
        title: "Saint Kitts and Nevis"
  Country_121:
    code: KP
    Translation:
      fr:
        title: "Corée du Nord"
      en:
        title: "North Korea"
  Country_122:
    code: KR
    Translation:
      fr:
        title: "Corée du Sud"
      en:
        title: "South Korea"
  Country_123:
    code: KW
    Translation:
      fr:
        title: "Koweït"
      en:
        title: "Kuwait"
  Country_124:
    code: KY
    Translation:
      fr:
        title: "Îles Caïmans"
      en:
        title: "Cayman Islands"
  Country_125:
    code: KZ
    Translation:
      fr:
        title: "Kazakhstan"
      en:
        title: "Kazakhstan"
  Country_126:
    code: LA
    Translation:
      fr:
        title: "Laos"
      en:
        title: "Laos"
  Country_127:
    code: LB
    Translation:
      fr:
        title: "Liban"
      en:
        title: "Lebanon"
  Country_128:
    code: LC
    Translation:
      fr:
        title: "Sainte-Lucie"
      en:
        title: "Saint Lucia"
  Country_129:
    code: LI
    Translation:
      fr:
        title: "Liechtenstein"
      en:
        title: "Liechtenstein"
  Country_130:
    code: LK
    Translation:
      fr:
        title: "Sri Lanka"
      en:
        title: "Sri Lanka"
  Country_131:
    code: LR
    Translation:
      fr:
        title: "Libéria"
      en:
        title: "Liberia"
  Country_132:
    code: LS
    Translation:
      fr:
        title: "Lesotho"
      en:
        title: "Lesotho"
  Country_133:
    code: LT
    Translation:
      fr:
        title: "Lituanie"
      en:
        title: "Lithuania"
  Country_134:
    code: LU
    Translation:
      fr:
        title: "Luxembourg"
      en:
        title: "Luxembourg"
  Country_135:
    code: LV
    Translation:
      fr:
        title: "Lettonie"
      en:
        title: "Latvia"
  Country_136:
    code: LY
    Translation:
      fr:
        title: "Libye"
      en:
        title: "Libya"
  Country_137:
    code: MA
    Translation:
      fr:
        title: "Maroc"
      en:
        title: "Morocco"
  Country_138:
    code: MC
    Translation:
      fr:
        title: "Monaco"
      en:
        title: "Monaco"
  Country_139:
    code: MD
    Translation:
      fr:
        title: "Moldavie"
      en:
        title: "Moldova"
  Country_140:
    code: ME
    Translation:
      fr:
        title: "Monténégro"
      en:
        title: "Montenegro"
  Country_141:
    code: MF
    Translation:
      fr:
        title: "Saint-Martin"
      en:
        title: "Saint Martin"
  Country_142:
    code: MG
    Translation:
      fr:
        title: "Madagascar"
      en:
        title: "Madagascar"
  Country_143:
    code: MH
    Translation:
      fr:
        title: "Îles Marshall"
      en:
        title: "Marshall Islands"
  Country_144:
    code: MK
    Translation:
      fr:
        title: "Macédoine"
      en:
        title: "Macedonia"
  Country_145:
    code: ML
    Translation:
      fr:
        title: "Mali"
      en:
        title: "Mali"
  Country_146:
    code: MM
    Translation:
      fr:
        title: "Myanmar"
      en:
        title: "Myanmar [Burma]"
  Country_147:
    code: MN
    Translation:
      fr:
        title: "Mongolie"
      en:
        title: "Mongolia"
  Country_148:
    code: MO
    Translation:
      fr:
        title: "R.A.S. chinoise de Macao"
      en:
        title: "Macau SAR China"
  Country_149:
    code: MP
    Translation:
      fr:
        title: "Îles Mariannes du Nord"
      en:
        title: "Northern Mariana Islands"
  Country_150:
    code: MQ
    Translation:
      fr:
        title: "Martinique"
      en:
        title: "Martinique"
  Country_151:
    code: MR
    Translation:
      fr:
        title: "Mauritanie"
      en:
        title: "Mauritania"
  Country_152:
    code: MS
    Translation:
      fr:
        title: "Montserrat"
      en:
        title: "Montserrat"
  Country_153:
    code: MT
    Translation:
      fr:
        title: "Malte"
      en:
        title: "Malta"
  Country_154:
    code: MU
    Translation:
      fr:
        title: "Maurice"
      en:
        title: "Mauritius"
  Country_155:
    code: MV
    Translation:
      fr:
        title: "Maldives"
      en:
        title: "Maldives"
  Country_156:
    code: MW
    Translation:
      fr:
        title: "Malawi"
      en:
        title: "Malawi"
  Country_157:
    code: MX
    Translation:
      fr:
        title: "Mexique"
      en:
        title: "Mexico"
  Country_158:
    code: MY
    Translation:
      fr:
        title: "Malaisie"
      en:
        title: "Malaysia"
  Country_159:
    code: MZ
    Translation:
      fr:
        title: "Mozambique"
      en:
        title: "Mozambique"
  Country_160:
    code: NA
    Translation:
      fr:
        title: "Namibie"
      en:
        title: "Namibia"
  Country_161:
    code: NC
    Translation:
      fr:
        title: "Nouvelle-Calédonie"
      en:
        title: "New Caledonia"
  Country_162:
    code: NE
    Translation:
      fr:
        title: "Niger"
      en:
        title: "Niger"
  Country_163:
    code: NF
    Translation:
      fr:
        title: "Île Norfolk"
      en:
        title: "Norfolk Island"
  Country_164:
    code: NG
    Translation:
      fr:
        title: "Nigéria"
      en:
        title: "Nigeria"
  Country_165:
    code: NI
    Translation:
      fr:
        title: "Nicaragua"
      en:
        title: "Nicaragua"
  Country_166:
    code: NL
    Translation:
      fr:
        title: "Pays-Bas"
      en:
        title: "Netherlands"
  Country_167:
    code: NO
    Translation:
      fr:
        title: "Norvège"
      en:
        title: "Norway"
  Country_168:
    code: NP
    Translation:
      fr:
        title: "Népal"
      en:
        title: "Nepal"
  Country_169:
    code: NR
    Translation:
      fr:
        title: "Nauru"
      en:
        title: "Nauru"
  Country_170:
    code: NU
    Translation:
      fr:
        title: "Niue"
      en:
        title: "Niue"
  Country_171:
    code: NZ
    Translation:
      fr:
        title: "Nouvelle-Zélande"
      en:
        title: "New Zealand"
  Country_172:
    code: OM
    Translation:
      fr:
        title: "Oman"
      en:
        title: "Oman"
  Country_173:
    code: PA
    Translation:
      fr:
        title: "Panama"
      en:
        title: "Panama"
  Country_174:
    code: PE
    Translation:
      fr:
        title: "Pérou"
      en:
        title: "Peru"
  Country_175:
    code: PF
    Translation:
      fr:
        title: "Polynésie française"
      en:
        title: "French Polynesia"
  Country_176:
    code: PG
    Translation:
      fr:
        title: "Papouasie-Nouvelle-Guinée"
      en:
        title: "Papua New Guinea"
  Country_177:
    code: PH
    Translation:
      fr:
        title: "Philippines"
      en:
        title: "Philippines"
  Country_178:
    code: PK
    Translation:
      fr:
        title: "Pakistan"
      en:
        title: "Pakistan"
  Country_179:
    code: PL
    Translation:
      fr:
        title: "Pologne"
      en:
        title: "Poland"
  Country_180:
    code: PM
    Translation:
      fr:
        title: "Saint-Pierre-et-Miquelon"
      en:
        title: "Saint Pierre and Miquelon"
  Country_181:
    code: PN
    Translation:
      fr:
        title: "Pitcairn"
      en:
        title: "Pitcairn Islands"
  Country_182:
    code: PR
    Translation:
      fr:
        title: "Porto Rico"
      en:
        title: "Puerto Rico"
  Country_183:
    code: PS
    Translation:
      fr:
        title: "Territoire palestinien"
      en:
        title: "Palestinian Territories"
  Country_184:
    code: PT
    Translation:
      fr:
        title: "Portugal"
      en:
        title: "Portugal"
  Country_185:
    code: PW
    Translation:
      fr:
        title: "Palaos"
      en:
        title: "Palau"
  Country_186:
    code: PY
    Translation:
      fr:
        title: "Paraguay"
      en:
        title: "Paraguay"
  Country_187:
    code: QA
    Translation:
      fr:
        title: "Qatar"
      en:
        title: "Qatar"
  Country_188:
    code: RE
    Translation:
      fr:
        title: "Réunion"
      en:
        title: "Réunion"
  Country_189:
    code: RO
    Translation:
      fr:
        title: "Roumanie"
      en:
        title: "Romania"
  Country_190:
    code: RS
    Translation:
      fr:
        title: "Serbie"
      en:
        title: "Serbia"
  Country_191:
    code: RU
    Translation:
      fr:
        title: "Russie"
      en:
        title: "Russia"
  Country_192:
    code: RW
    Translation:
      fr:
        title: "Rwanda"
      en:
        title: "Rwanda"
  Country_193:
    code: SA
    Translation:
      fr:
        title: "Arabie saoudite"
      en:
        title: "Saudi Arabia"
  Country_194:
    code: SB
    Translation:
      fr:
        title: "Îles Salomon"
      en:
        title: "Solomon Islands"
  Country_195:
    code: SC
    Translation:
      fr:
        title: "Seychelles"
      en:
        title: "Seychelles"
  Country_196:
    code: SD
    Translation:
      fr:
        title: "Soudan"
      en:
        title: "Sudan"
  Country_197:
    code: SE
    Translation:
      fr:
        title: "Suède"
      en:
        title: "Sweden"
  Country_198:
    code: SG
    Translation:
      fr:
        title: "Singapour"
      en:
        title: "Singapore"
  Country_199:
    code: SH
    Translation:
      fr:
        title: "Sainte-Hélène"
      en:
        title: "Saint Helena"
  Country_200:
    code: SI
    Translation:
      fr:
        title: "Slovénie"
      en:
        title: "Slovenia"
  Country_201:
    code: SJ
    Translation:
      fr:
        title: "Svalbard et Île Jan Mayen"
      en:
        title: "Svalbard and Jan Mayen"
  Country_202:
    code: SK
    Translation:
      fr:
        title: "Slovaquie"
      en:
        title: "Slovakia"
  Country_203:
    code: SL
    Translation:
      fr:
        title: "Sierra Leone"
      en:
        title: "Sierra Leone"
  Country_204:
    code: SM
    Translation:
      fr:
        title: "Saint-Marin"
      en:
        title: "San Marino"
  Country_205:
    code: SN
    Translation:
      fr:
        title: "Sénégal"
      en:
        title: "Senegal"
  Country_206:
    code: SO
    Translation:
      fr:
        title: "Somalie"
      en:
        title: "Somalia"
  Country_207:
    code: SR
    Translation:
      fr:
        title: "Suriname"
      en:
        title: "Suriname"
  Country_208:
    code: ST
    Translation:
      fr:
        title: "Sao Tomé-et-Principe"
      en:
        title: "São Tomé and Príncipe"
  Country_209:
    code: SV
    Translation:
      fr:
        title: "El Salvador"
      en:
        title: "El Salvador"
  Country_210:
    code: SY
    Translation:
      fr:
        title: "Syrie"
      en:
        title: "Syria"
  Country_211:
    code: SZ
    Translation:
      fr:
        title: "Swaziland"
      en:
        title: "Swaziland"
  Country_212:
    code: TC
    Translation:
      fr:
        title: "Îles Turks et Caïques"
      en:
        title: "Turks and Caicos Islands"
  Country_213:
    code: TD
    Translation:
      fr:
        title: "Tchad"
      en:
        title: "Chad"
  Country_214:
    code: TF
    Translation:
      fr:
        title: "Terres australes françaises"
      en:
        title: "French Southern Territories"
  Country_215:
    code: TG
    Translation:
      fr:
        title: "Togo"
      en:
        title: "Togo"
  Country_216:
    code: TH
    Translation:
      fr:
        title: "Thaïlande"
      en:
        title: "Thailand"
  Country_217:
    code: TJ
    Translation:
      fr:
        title: "Tadjikistan"
      en:
        title: "Tajikistan"
  Country_218:
    code: TK
    Translation:
      fr:
        title: "Tokelau"
      en:
        title: "Tokelau"
  Country_219:
    code: TL
    Translation:
      fr:
        title: "Timor oriental"
      en:
        title: "Timor-Leste"
  Country_220:
    code: TM
    Translation:
      fr:
        title: "Turkménistan"
      en:
        title: "Turkmenistan"
  Country_221:
    code: TN
    Translation:
      fr:
        title: "Tunisie"
      en:
        title: "Tunisia"
  Country_222:
    code: TO
    Translation:
      fr:
        title: "Tonga"
      en:
        title: "Tonga"
  Country_223:
    code: TR
    Translation:
      fr:
        title: "Turquie"
      en:
        title: "Turkey"
  Country_224:
    code: TT
    Translation:
      fr:
        title: "Trinité-et-Tobago"
      en:
        title: "Trinidad and Tobago"
  Country_225:
    code: TV
    Translation:
      fr:
        title: "Tuvalu"
      en:
        title: "Tuvalu"
  Country_226:
    code: TW
    Translation:
      fr:
        title: "Taïwan"
      en:
        title: "Taiwan"
  Country_227:
    code: TZ
    Translation:
      fr:
        title: "Tanzanie"
      en:
        title: "Tanzania"
  Country_228:
    code: UA
    Translation:
      fr:
        title: "Ukraine"
      en:
        title: "Ukraine"
  Country_229:
    code: UG
    Translation:
      fr:
        title: "Ouganda"
      en:
        title: "Uganda"
  Country_230:
    code: UM
    Translation:
      fr:
        title: "Îles Mineures Éloignées des États-Unis"
      en:
        title: "U.S. Minor Outlying Islands"
  Country_231:
    code: US
    Translation:
      fr:
        title: "États-Unis"
      en:
        title: "United States"
  Country_232:
    code: UY
    Translation:
      fr:
        title: "Uruguay"
      en:
        title: "Uruguay"
  Country_233:
    code: UZ
    Translation:
      fr:
        title: "Ouzbékistan"
      en:
        title: "Uzbekistan"
  Country_234:
    code: VA
    Translation:
      fr:
        title: "État de la Cité du Vatican"
      en:
        title: "Vatican City"
  Country_235:
    code: VC
    Translation:
      fr:
        title: "Saint-Vincent-et-les Grenadines"
      en:
        title: "Saint Vincent and the Grenadines"
  Country_236:
    code: VE
    Translation:
      fr:
        title: "Venezuela"
      en:
        title: "Venezuela"
  Country_237:
    code: VG
    Translation:
      fr:
        title: "Îles Vierges britanniques"
      en:
        title: "British Virgin Islands"
  Country_238:
    code: VI
    Translation:
      fr:
        title: "Îles Vierges des États-Unis"
      en:
        title: "U.S. Virgin Islands"
  Country_239:
    code: VN
    Translation:
      fr:
        title: "Viêt Nam"
      en:
        title: "Vietnam"
  Country_240:
    code: VU
    Translation:
      fr:
        title: "Vanuatu"
      en:
        title: "Vanuatu"
  Country_241:
    code: WF
    Translation:
      fr:
        title: "Wallis-et-Futuna"
      en:
        title: "Wallis and Futuna"
  Country_242:
    code: WS
    Translation:
      fr:
        title: "Samoa"
      en:
        title: "Samoa"
  Country_243:
    code: YE
    Translation:
      fr:
        title: "Yémen"
      en:
        title: "Yemen"
  Country_244:
    code: YT
    Translation:
      fr:
        title: "Mayotte"
      en:
        title: "Mayotte"
  Country_245:
    code: ZA
    Translation:
      fr:
        title: "Afrique du Sud"
      en:
        title: "South Africa"
  Country_246:
    code: ZM
    Translation:
      fr:
        title: "Zambie"
      en:
        title: "Zambia"
  Country_247:
    code: ZW
    Translation:
      fr:
        title: "Zimbabwe"
      en:
        title: "Zimbabwe"
  Country_248:
    code: ZZ
    Translation:
      fr:
        title: "région indéterminée"
      en:
        title: "Unknown or Invalid Region"

Lately I have been working a lot with YAML data in my HAML templates via Middleman. YAML is great for providing data to display on your site. It allows for easy modification and addition trough separate .yml files instead of having to copy-paste HTML/HAML. YAML is very intuitive to write and work with from Ruby (and templates). But if you are in need of some good documentation, check out this YAML Cookbook.

Rails 2->3 Upgrade Part 2

Ok right off the bat, acts_as_archive… I’m going to say s implementation kind of sucks. I appreciate that they let you use a config file to configure your archive tables…but it doesn’t work. I found a branch that had a fix, but I’m still going to have to update my app code for the new api.

Next, what is up with YAML? You got native yaml, Syck, now Psych? sheesh can we just settle on one please? Anyways, Psych is supposedly our new Leader so make sure all yml’s conform to His/Her One True Structure.

What else…oh yeah cells. Another huge API change here…I’ll be back in a few days once that is all updated and tested.

#yaml #memo - Comparing with JSON & XML (Part 2)

Technically YAML is a superset of JSON. This means that, in theory at least, a YAML parser can understand JSON, but not necessarily the other way around.

See the official specs, in the section entitled "YAML: Relation to JSON".

In general, there are certain things I like about YAML that are not available in JSON.

  • As @jdupont pointed out, YAML is visually easier to look at. In fact the YAML homepage is itself valid YAML, yet it is easy for a human to read.
  • YAML has the ability to reference other items within a YAML file using “anchors.” Thus it can handle relational information as one might find in a MySQL database.
  • YAML is more robust about embedding other serialization formats such as JSON or XML within a YAML file.

In practice neither of these last two points will likely matter for things that you or I do, but in the long term, I think YAML will be a more robust and viable data serialization format.

Right now, AJAX and other web technologies tend to use JSON. YAML is currently being used more for offline data processes. For example, it is included by default in the C-based OpenCV computer vision package, whereas JSON is not.

You will find C libraries for both JSON and YAML. YAML’s libraries tend to be newer, but I have had no trouble with them in the past. See for example Yaml-cpp.

Symfony2 xml to yaml snippet #2
XML: <services> <service class="%sonata.user.admin.user.class%"> <tag name="sonata.admin" manager_type="orm" group="%sonata.user.admin.groupname%" label="%sonata.user.admin.user.label%" /> <argument /> <argument>%sonata.user.admin.user.entity%</argument> <argument /> <call method="setUserManager"> <argument type="service" /> </call> </service> </services> Turns to YAML: services: sonata.user.admin.user: class: %sonata.user.admin.user.class% tags: - { name: sonata.admin, manager_type: orm, group: %sonata.user.admin.groupname%, label: %sonata.user.admin.user.label% } arguments: [null, %sonata.user.admin.user.entity%, null] calls: - [setUserManager, [@fos_user.user_manager]]
routing.yml の requirements での正規表現について - symfony1.4

ドキュメントにも載っているやり方。

hoge:
  url: /hoge/:name
  param: { module: hoge, action: fuga }
  requirements: { name: \w+, sf_method: get }

特定のワードのみ許可。

hoge:
  url: /hoge/:name
  param: { module: hoge, action: fuga }
  requirements: { name: (unko|yeah), sf_method: get }

自由に正規表現を書きたい場合。

hoge:
  url: /hoge/:name
  param: { module: hoge, action: fuga }
  requirements: { name: '^(unko|yeah)-[a-z_-]+', sf_method: get }
XML pra quê? Conheça o JSON e o YAML

Por muito tempo o XML dominou a web quando o assunto era serialização de dados. Aos poucos, outras linguagens foram se popularizando e conquistando desenvolvedores por conta das fraquezas desse padrão recomendado pela W3C.

Enquanto isso, poderosas redes sociais, como o Twitter, foram adotando linguagens como o JSON ao fornecer dados através de suas APIs. E com isso, tomando grandes proporções no mercado. 

Vamos entender sobre o que cada uma se trata, antes de entrarmos mais a fundo no assunto.

  • JSON (JavaScript Object Notation) é uma estrutura de dados baseada em javascript. Apesar de estar na “moda” agora, não é uma tecnologia nova, faz parte do javascript puro e não precisa de nenhum script pra trabalhar.
  • YAML acrônimo recursivo para YAML Ain’t Markup Language. No início do seu desenvolvimento significava Yet Another Markup Language (Mais outra linguagem de marcação) para distinguir seu propósito centrado em dados no lugar de documentos marcados. Foi projetada para “ser legível aos seres-humanos”.

Vejamos, então, um exemplo de uma estrutura de dados usando JSON.

{
"nome": "zeno rocha - blog",
"url": "http://blog.zenorocha.com",
"posts": [
{
"titulo": "do it! - as simple as you can",
"data": "13/04/2011"
},
{
"titulo": hello world,
"data": "12/04/2011"
}
]
}

A mesma estrutura utilizando YAML:

blog:
nome: zeno rocha - blog
url: http://blog.zenorocha.com
post:
titulo: do it! - as simple as you can
data: 13/04/2011
post:
titulo: hello world
data: 12/04/2011

E agora compare-as com o formato XML:

<?xml version="1.0" encoding="utf-8"?>
<blog>
<nome>zeno rocha - blog</nome>
<url>http://blog.zenorocha.com</url>
<post>
<titulo>do it! - as simple as you can</titulo>
<data>13/04/2011</data>
</post>
<post>
<titulo>hello world</titulo>
<data>12/04/2011</data>
</post>
</blog>

Se você está acostumado com linguagens de marcação como HTML pode até discordar de mim, mas em termos de leitura o XML é muito menos intuitivo do que as outras linguagens apresentadas.

Além disso, a estrutura de dados é mais simples de trabalhar e o tempo de execução de um script lendo dados em JSON é dezenas de vezes mais rápido do que ler um conteúdo XML.

As principais linguagens de programação server side tem suporte pra escrever dados em JSON. Mais do que isso, existem métodos nativos para decodificação desse padrão.

O JSON pode substituir sim o XML, e faz isso muito bem na hora de trabalhar com respostas em AJAX por exemplo. Entretanto, seria muita pretensão achar que JSON irá desbancar um formato tão estável em tão pouco tempo.

Don't expose your keys and why Amazon is awesome!

So I had a weird number pop up on my SkypeIn today and answered. It was a pretty cool sounding dude from Amazon called Ty. He said he’d sent me an email about a security breach on my account and hadn’t heard anything back.

I remember getting a cryptically worded email from Amazon about this yesterday but the instructions in the email didn’t match the updated UI of AWS so I got a bit lost trying to sort the problem out. I made sure I reset the password though and sent a reply to the email asking them to send updated instructions.

Read More

Travis CI 入門:GitHub + Travis CI で継続的インテグレーション

1. Travis CI とは

Travis CI はオープンソースコミュニティのためにホストされた CI(継続的インテグレーション)サービスです。

継続的インテグレーションってなんだ?

継続的インテグレーション、CI(英: continuous integration)とは、主にプログラマーのアプリケーション作成時の品質改善や納期の短縮のための習慣のことである。エクストリーム・プログラミング (XP) のプラクティスの一つで、狭義にはビルドやテスト、インスペクションなどを継続的に実行していくことを意味する。特に、近年の開発においては、継続的インテグレーションをサポートするソフトウェアを使用することがある。

引用: 継続的インテグレーション - Wikipedia

Travis CI は GitHub と連携しており、CI したいリポジトリーを接続しておくと、Travis CI がコミットを取得して、設定ファイル通りにビルド・テストしてくれます。そしてビルド・テストに失敗するとメール(他も可)で結果が送られてきます。

以下のような流れでビルド・テストが行われます。

 +---> [Developer] --- (1) ---> [GitHub] <---+
 |                                           |
 +----- (3) ----- [Travis CI] ----- (2) -----+

(1) ソースコミット
(2) 未テストのコミット取得
(3) テスト結果を送信

Read More

Text
Photo
Quote
Link
Chat
Audio
Video