Valk Posted January 19, 2017 Author Posted January 19, 2017 Yep the customize_image.sh script set emmc_fix=on fine. Be awesome if you could only needing to plugin a usb stick would be a nice setup, that said I'm also grateful that I've been forced to learn how it works instead of trusting magic scripts . Ideally when I get this ready for production I still would prefer to install via Ethernet but USB is still a big improvement
Valk Posted January 19, 2017 Author Posted January 19, 2017 Ok well installed via dd root@clearfogpro~# dd if=Armbian_5.24_Clearfogpro_Debian_jessie_next_4.9.4.img 2478080+0 records in 2478080+0 records out 1268776960 bytes (1.3 GB) copied, 1[ 299.354823] mmcblk0: p1 89.472 s, 6.7 MB/s root@clearfogpro:~# sync root@clearfogpro:~# echo 0 > /sys/block/mmcblk0boot0/force_ro root@clearfogpro:~# ifconfig eth0 192.168.0.30 root@clearfogpro:~# ifconfig eth0 netmask 255.255.255.0 root@clearfogpro:~# scp yocto@192.168.0.250:/home/yocto/4.9/u-boot.mmc ./ yocto@192.168.0.250's password: u-boot.mmc 100% 935KB 934.6KB/s 00:00 root@clearfogpro:~# dd if=u-boot.mmc of=/dev/mmcblk0boot0 1869+1 records in 1869+1 records out 957048 bytes (957 kB) copied, 0.143723 s, 6.7 MB/s root@clearfogpro:~# echo 1 > /sys/block/mmcblk0boot0/force_ro root@clearfogpro:~# sync root@clearfogpro:~# mount /dev/mmcblk0p1 /media/emmc/ [ 355.858748] EXT4-fs (mmcblk0p1): mounted filesystem with writeback data mode. Opts: (null) root@clearfogpro:~# umount /media/emmc root@clearfogpro:~# poweroff P.S. Forgot to write bs=4M to speed it up Wrote u-boot again because I didn't want to have to restore from uart if it was corrupted. Had it stall again at Starting Kernel ... but I turned it off and let it cool down. Fired it up again and it started working. Not sure what's happening there, need to give solidrun an e-mail.
zador.blood.stained Posted January 19, 2017 Posted January 19, 2017 So it worked for you in the end if I understand it correctly? I'm not sure you need to touch emmc boot partition at all (unless solid-run instructions tell you to do so) because u-boot should be loaded from its usual place between the partition table and the first partition. Also your microSOM board may contain SPI flash so you could write the u-boot to it and load it after that just by fiddling with the onboard DIP switches, but it's not enabled by default in the next kernel branch, I'll add a patch later.
Valk Posted January 19, 2017 Author Posted January 19, 2017 Yes booting the system properly meant that setting the emmc_fix flag on worked as expected. And yes I didn't need to touch the boot partition in the end. As for the SPI flash you've got me thinking. There's a dip switch for it just need to check where the pins are on the PCB. Hope that's an option because it'll make things much easier. Unfortunately I'm having even more troubles. After installing the system I do an update and install a few packages + hardware (wifi, hostapd, and a few others). I use a script I've had for a while to do the install but unfortunately when it reboots the system it fails to read from the emmc (this .scr file). Hopefully solid-run get back to me with a few details as to what's going on. I've even tried slowing the CPU to 800Mhz with no luck.
zador.blood.stained Posted January 19, 2017 Posted January 19, 2017 As for the SPI flash you've got me thinking. There's a dip switch for it just need to check where the pins are on the PCB. Hope that's an option because it'll make things much easier. On my board the SPI flash is already there - on the bottom of the microSOM, so you can't see it unless you remove the heatsink and disconnect the microSOM from the carrier board, but according to Solid-Run docs it is optional.
zador.blood.stained Posted January 19, 2017 Posted January 19, 2017 Just added improved USB boot support Booting from SPI flash+USB will work out of the box, BootROM - 1.73 Booting from SPI flash General initialization - Version: 1.0.0 Detected Device ID 6828 High speed PHY - Version: 2.0 Init Customer board board SerDes lanes topology details: | Lane # | Speed| Type | ------------------------------| | 0 | 3 | SATA0 | | 1 | 0 | SGMII1 | | 2 | 5 | PCIe1 | | 3 | 5 | USB3 HOST1 | | 4 | 5 | PCIe2 | | 5 | 0 | SGMII2 | ------------------------------- PCIe, Idx 1: detected no link PCIe, Idx 2: detected no link High speed PHY - Ended Successfully DDR3 Training Sequence - Ver TIP-1.39.0 DDR3 Training Sequence - Switching XBAR Window to FastPath Window DDR3 Training Sequence - Ended Successfully BootROM: Image checksum verification PASSED __ __ _ _ | \/ | __ _ _ ____ _____| | | | |\/| |/ _` | '__\ \ / / _ \ | | | | | | (_| | | \ V / __/ | | |_| |_|\__,_|_| \_/ \___|_|_| _ _ ____ _ | | | | | __ ) ___ ___ | |_ | | | |___| _ \ / _ \ / _ \| __| | |_| |___| |_) | (_) | (_) | |_ \___/ |____/ \___/ \___/ \__| ** LOADER ** U-Boot 2013.01 (Jan 19 2017 - 15:31:46) Marvell version: 2015_T1.0p11 Board: A38x-Customer-Board-1 SoC: MV88F6828 Rev A0 running 2 CPUs CPU: ARM Cortex A9 MPCore (Rev 1) LE CPU 0 CPU @ 1600 [MHz] L2 @ 800 [MHz] TClock @ 250 [MHz] DDR3 @ 800 [MHz] DDR3 32 Bit Width,FastPath Memory Access, DLB Enabled, ECC Disabled DRAM: 1 GiB MMC: mv_sdh: 0 *** Warning - bad CRC, using default environment PCI-e 0: Detected No Link. PCI-e 1: Detected No Link. USB2.0 0: Host Mode USB3.0 0: Host Mode USB3.0 1: Host Mode Map: Code: 0x3fecf000:0x3ff97a30 BSS: 0x3ffef81c Stack: 0x3f9aef20 Heap: 0x3f9af000:0x3fecf000 U-Boot Environment: 0x00100000:0x00120000 Board configuration detected: Net: | port | Interface | PHY address | |--------|-----------|--------------| | egiga0 | RGMII | 0x00 | | egiga1 | SGMII | In-Band | | egiga2 | SGMII | In-Band | egiga0 [PRIME], egiga1, egiga2 Hit any key to stop autoboot: 0 Please store the boot environment on the NOR SPI flash to override the default boot sequence Trying to boot from USB (Re)start USB... USB0: Port (usbActive) : 1 Interface (usbType = 3) : USB XHCI 1.00 scanning bus 0 for devices... 2 USB Device(s) found scanning usb for storage devices... 1 Storage Device(s) found scanning usb for ethernet devices... 0 Ethernet Device(s) found 1838 bytes read in 224 ms (7.8 KiB/s) ## Executing script at 03000000 Boot script loaded from usb 172 bytes read in 188 ms (0 Bytes/s) 18192 bytes read in 423 ms (42 KiB/s) 8141983 bytes read in 575 ms (13.5 MiB/s) 4825872 bytes read in 487 ms (9.4 MiB/s) ## Loading init Ramdisk from Legacy Image at 02880000 ... Image Name: uInitrd Created: 2017-01-19 12:10:59 UTC Image Type: ARM Linux RAMDisk Image (gzip compressed) Data Size: 8141919 Bytes = 7.8 MiB Load Address: 00000000 Entry Point: 00000000 ## Flattened Device Tree blob at 02040000 Booting using the fdt blob at 0x02040000 Using Device Tree in place at 02040000, end 0204770f Skipping Device Tree update ('fdt_skip_update' = yes) Limit DDR size at 3GB due to power of 2 requirement of Address decoding Starting kernel ... Uncompressing Linux... done, booting the kernel. Loading, please wait... starting version 229 Begin: Loading essential drivers ... done. Begin: Running /scripts/init-premount ... done. Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done. Begin: Running /scripts/local-premount ... Scanning for Btrfs filesystems done. Begin: Will now check root file system ... fsck from util-linux 2.27.1 [/sbin/fsck.ext4 (1) -- /dev/sda1] fsck.ext4 -a -C0 /dev/sda1 /dev/sda1: clean, 49617/75840 files, 213306/303104 blocks done. done. Begin: Running /scripts/local-bottom ... done. Begin: Running /scripts/init-bottom ... done. Welcome to Ubuntu 16.04.1 LTS! Booting from UART+USB should work like this: Write the Armbian image to the USB flash drive Connect it to the USB3.0 port Load the u-boot from UART (or from SD/eMMC assuming you got the latest version there) Stop the default boot sequence Execute in u-boot prompt: run usbboot Observe the result Marvell>> run usbboot (Re)start USB... USB0: Port (usbActive) : 1 Interface (usbType = 3) : USB XHCI 1.00 scanning bus 0 for devices... 2 USB Device(s) found scanning usb for storage devices... 1 Storage Device(s) found scanning usb for ethernet devices... 0 Ethernet Device(s) found 1838 bytes read in 204 ms (8.8 KiB/s) ## Executing script at 03000000 Boot script loaded from usb 172 bytes read in 178 ms (0 Bytes/s) 18192 bytes read in 423 ms (42 KiB/s) 8141983 bytes read in 554 ms (14 MiB/s) 4825872 bytes read in 471 ms (9.8 MiB/s) ## Loading init Ramdisk from Legacy Image at 02880000 ... Image Name: uInitrd Created: 2017-01-19 12:10:59 UTC Image Type: ARM Linux RAMDisk Image (gzip compressed) Data Size: 8141919 Bytes = 7.8 MiB Load Address: 00000000 Entry Point: 00000000 ## Flattened Device Tree blob at 02040000 Booting using the fdt blob at 0x02040000 Using Device Tree in place at 02040000, end 0204770f Skipping Device Tree update ('fdt_skip_update' = yes) Limit DDR size at 3GB due to power of 2 requirement of Address decoding Starting kernel ... Uncompressing Linux... done, booting the kernel. Loading, please wait... starting version 229 Begin: Loading essential drivers ... done. Begin: Running /scripts/init-premount ... done. Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done. Begin: Running /scripts/local-premount ... Scanning for Btrfs filesystems done. Begin: Will now check root file system ... fsck from util-linux 2.27.1 [/sbin/fsck.ext4 (1) -- /dev/sda1] fsck.ext4 -a -C0 /dev/sda1 /dev/sda1: clean, 49636/894912 files, 311677/3841216 blocks done. done. Begin: Running /scripts/local-bottom ... done. Begin: Running /scripts/init-bottom ... done. Welcome to Ubuntu 16.04.1 LTS!
Valk Posted January 19, 2017 Author Posted January 19, 2017 I was actually looking at the pcb schematic to find the SPI port. Unfortunately if I have to pull apart the heatsink then I'm better off via uart (which is fine). Still "run usbboot" nice touch Thanks for doing that so quick. Hopefully I can get to the bottom of the emmc issues soon. Thanks for all the help.
zador.blood.stained Posted January 19, 2017 Posted January 19, 2017 I was actually looking at the pcb schematic to find the SPI port. Unfortunately if I have to pull apart the heatsink then I'm better off via uart (which is fine). Still "run usbboot" nice touch Just please build an image for the Clearfog Base with default kernel, boot it (it should boot on Pro too) and check dmesg for something like this: root@clearfogbase:~# dmesg | grep -i spi [ 0.880393] m25p80 spi1.0: w25q32 (4096 Kbytes) [ 0.880439] 3 ofpart partitions found on MTD device spi1.0 [ 0.883832] Creating 3 MTD partitions on "spi1.0": and for /dev/mtd* devices. You don't need to disassembe anything - SPI flash presence should be detected programmatically
Valk Posted January 19, 2017 Author Posted January 19, 2017 Hmm well usb boot works wonderfully. Got a feeling my problems might be to do with the resize script I can: - boot to USB- dd the system image over- boot the emmc system and do login sequence if I immediately reboot the system (as the system recommends) it fails to boot from emmc with: Trying to boot from MMCFailed to mount ext2 filesystem...** Unrecognized filesystem type **Failed to mount ext2 filesystem...** Unrecognized filesystem type ** [ OK ] Reached target Shutdown. [ 93.638226] reboot: Restarting system BootROM - 1.73 Booting from MMC General initialization - Version: 1.0.0 Detected Device ID 6828 High speed PHY - Version: 2.0 Init Customer board board SerDes lanes topology details: | Lane # | Speed| Type | ------------------------------| | 0 | 3 | SATA0 | | 1 | 0 | SGMII1 | | 2 | 5 | PCIe1 | | 3 | 5 | USB3 HOST1 | | 4 | 5 | PCIe2 | | 5 | 0 | SGMII2 | ------------------------------- PCIe, Idx 1: detected no link PCIe, Idx 2: detected no link High speed PHY - Ended Successfully DDR3 Training Sequence - Ver TIP-1.39.0 DDR3 Training Sequence - Switching XBAR Window to FastPath Window DDR3 Training Sequence - Ended Successfully BootROM: Image checksum verification PASSED __ __ _ _ | \/ | __ _ _ ____ _____| | | | |\/| |/ _` | '__\ \ / / _ \ | | | | | | (_| | | \ V / __/ | | |_| |_|\__,_|_| \_/ \___|_|_| _ _ ____ _ | | | | | __ ) ___ ___ | |_ | | | |___| _ \ / _ \ / _ \| __| | |_| |___| |_) | (_) | (_) | |_ \___/ |____/ \___/ \___/ \__| ** LOADER ** U-Boot 2013.01 (Jan 20 2017 - 07:14:14) Marvell version: 2015_T1.0p11 Board: A38x-Customer-Board-1 SoC: MV88F6828 Rev A0 running 2 CPUs CPU: ARM Cortex A9 MPCore (Rev 1) LE CPU 0 CPU @ 666 [MHz] L2 @ 333 [MHz] TClock @ 250 [MHz] DDR3 @ 333 [MHz] DDR3 32 Bit Width,FastPath Memory Access, DLB Enabled, ECC Disabled DRAM: 1 GiB MMC: mv_sdh: 0 sdhci_transfer_data: Error detected in status(0x408000)! *** Warning - bad CRC, using default environment PCI-e 0: Detected No Link. PCI-e 1: Detected No Link. USB2.0 0: Host Mode USB3.0 0: Host Mode USB3.0 1: Host Mode Map: Code: 0x3fed2000:0x3ff98134 BSS: 0x3ffefd5c Stack: 0x3f9c1f20 Heap: 0x3f9c2000:0x3fed2000 U-Boot Environment: 0x000f0000:0x00100000 (MMC) Board configuration detected: Net: | port | Interface | PHY address | |--------|-----------|--------------| | egiga0 | RGMII | 0x00 | | egiga1 | SGMII | In-Band | | egiga2 | SGMII | In-Band | egiga0 [PRIME], egiga1, egiga2 Hit any key to stop autoboot: 0 Trying to boot from MMC Failed to mount ext2 filesystem... ** Unrecognized filesystem type ** Failed to mount ext2 filesystem... ** Unrecognized filesystem type ** ## Executing script at 03000000 Boot script loaded from mmc Failed to mount ext2 filesystem... ** Unrecognized filesystem type ** Failed to mount ext2 filesystem... ** Unrecognized filesystem type ** Failed to mount ext2 filesystem... ** Unrecognized filesystem type ** Failed to mount ext2 filesystem... ** Unrecognized filesystem type ** Failed to mount ext2 filesystem... ** Unrecognized filesystem type ** Failed to mount ext2 filesystem... ** Unrecognized filesystem type ** Failed to mount ext2 filesystem... ** Unrecognized filesystem type ** Failed to mount ext2 filesystem... ** Unrecognized filesystem type ** Bad Linux ARM zImage magic! Trying to boot from USB (Re)start USB... USB0: Port (usbActive) : 1 Interface (usbType = 3) : USB XHCI 1.00 scanning bus 0 for devices... 1 USB Device(s) found scanning usb for storage devices... 0 Storage Device(s) found scanning usb for ethernet devices... 0 Ethernet Device(s) found ** Bad device usb 0 ** ** Bad device usb 0 ** ## Executing script at 03000000 Boot script loaded from usb ** Bad device usb 0 ** ** Bad device usb 0 ** ** Bad device usb 0 ** ** Bad device usb 0 ** ** Bad device usb 0 ** ** Bad device usb 0 ** ** Bad device usb 0 ** ** Bad device usb 0 ** Bad Linux ARM zImage magic! Default boot sequence failed - falling back to TFTP Using egiga0 device TFTP from server 10.4.50.38; our IP address is 10.4.50.170 Filename 'uImage'. Load address: 0x2000000 Loading: T T Abort Using egiga0 device TFTP from server 10.4.50.38; our IP address is 10.4.50.170 Filename 'armada-388-clearfog-base.dtb'. Load address: 0x2000000 Loading: * Abort Bad Linux ARM zImage magic! Marvell>>
Valk Posted January 19, 2017 Author Posted January 19, 2017 as for spi I get nothing from dmesg and have no mtd devices in /dev
zador.blood.stained Posted January 19, 2017 Posted January 19, 2017 Hm. FS resizing should not require the reboot in the first place (and it works without it when booting from the SD). Must be something eMMC specific that breaks the process. Please try to extract /var/log/armhwinfo.log after the first reboot (or "sudo armbianmonitor -u" should upload it and provide the link to it automatically) Edit: Hm, I wonder if it counts eMMC boot partitions or device size somewhere so parted/fdisk produce unexpected output...
Valk Posted January 19, 2017 Author Posted January 19, 2017 While I get it ready as a quick side note. Now when the system boots the leds for Ethernet ports on the switch stay on all the time.
Valk Posted January 19, 2017 Author Posted January 19, 2017 Ok got the log file: http://sprunge.us/CjIS
Valk Posted January 19, 2017 Author Posted January 19, 2017 ah sorry after the first reboot, one moment
Valk Posted January 19, 2017 Author Posted January 19, 2017 Ok a little difficult. Do you know how I can repair the partition enough to get the file off it? root@clearfogpro:~# ls root@clearfogpro:~# ls /media/ root@clearfogpro:~# mkdir /media/emmc root@clearfogpro:~# mount /dev/mmcblk0p1 /media/emmc/ [ 243.366150] UDF-fs: warning (device mmcblk0p1): udf_fill_super: No partition found (2) [ 243.377055] F2FS-fs (mmcblk0p1): Magic Mismatch, valid(0xf2f52010) - read(0x1102d4b) [ 243.384864] F2FS-fs (mmcblk0p1): Can't find valid F2FS filesystem in 1th superblock [ 243.394112] F2FS-fs (mmcblk0p1): Magic Mismatch, valid(0xf2f52010) - read(0x6b636170) [ 243.401972] F2FS-fs (mmcblk0p1): Can't find valid F2FS filesystem in 2th superblock [ 243.409723] F2FS-fs (mmcblk0p1): Magic Mismatch, valid(0xf2f52010) - read(0x1102d4b) [ 243.417513] F2FS-fs (mmcblk0p1): Can't find valid F2FS filesystem in 1th superblock [ 243.425211] F2FS-fs (mmcblk0p1): Magic Mismatch, valid(0xf2f52010) - read(0x6b636170) [ 243.433095] F2FS-fs (mmcblk0p1): Can't find valid F2FS filesystem in 2th superblock mount: wrong fs type, bad option, bad superblock on /dev/mmcblk0p1, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so.
zador.blood.stained Posted January 20, 2017 Posted January 20, 2017 ah sorry after the first reboot, one moment Sorry, not after the first reboot, but before it, so the log you provided is fine. As I suspected, looks like additional eMMC partitions cause incorrect behavior of the resize procedure, so I'll try to rework it.
zador.blood.stained Posted January 22, 2017 Posted January 22, 2017 Resizing procedure should be (hopefully) fixed, and also I have to correct myself - according to the wiki u-boot can be installed either to the main eMMC storage or to either one of its boot partitions (with unknown priority).
Valk Posted January 22, 2017 Author Posted January 22, 2017 Ok well that's annoying... Talking with Solidrun they insisted I try their compiled version of debian to help with resolving the issues. I haven't got back to them yet but I can't fault the solidrun image. I'm at the situation now where I'm using the solidrun u-boot and trying to boot the system via tftp: setenv serverip 192.168.1.240setenv ipaddr 192.168.1.1setenv fdt_high 0x07a12000 ( or setenv fdt_high 0xffffffff ) setenv bootargs "selinux=0 cgroup_disable=memory scandelay root=/dev/sda1 rw rootfstype=ext4 console=ttyS0,115200 loglevel=7 rootwait" tftp ${fdtaddr} 4.9/armada-388-clearfog.dtbtftp ${loadaddr} 4.9/vmlinuz-4.9.4-mvebubootz ${loadaddr} - ${fdtaddr} The system will stall at Starting kernel .... after I can hit the hardware reboot and let the emmc boot (solidrun official image) and it loads fine. https://images.solid-build.xyz/A38X/sr-a38x-debian-jessie-20161101-testing.img.xz https://images.solid-build.xyz/A38X/u-boot.mmc Considering the system is using the solidrun u-boot that crosses it off the list (not that there's really any difference between the armbian one except it has a few nice tweaks) So that leaves the an issue with the dtb file or vmlinuz image. Just went through the dts between the solidrun repo: https://github.com/SolidRun/linux-stable/tree/linux-4.4.y-clearfog/arch/arm/boot/dts and I can't see any meaningful changes. Leaves the last part the image it boots and that's a much bigger needle to solve. Do you experience any issues booting the system if it has been running for say 30 minutes? Going to try the 4.9.5 branch which I just compiled then try specifying the solidrun repo https://github.com/SolidRun/linux-stable/tree/linux-4.4.y-clearfog and see how they go. Really want to get it going with the current kernel stable so the system can be easily kept up to date in the future.
Valk Posted January 22, 2017 Author Posted January 22, 2017 -- Deleted -- Thought SolidRun's image was faulty but I think I might have broken it so restoring their system image. On a side note I used the 3.10.104 (4.9.5 wouldn't boot) armbian kernel image to get the dtb and vmlinuz files to boot and they managed to boot fine (haven't reset the device). I'm thinking there's a software issue during boot.
Igor Posted January 23, 2017 Posted January 23, 2017 Our default recommendation is also Debian with kernel 4.4.x (as written on download page) since it's currently most polished kernel and also supported by Solidrun too.
zador.blood.stained Posted January 23, 2017 Posted January 23, 2017 Do you experience any issues booting the system if it has been running for say 30 minutes? No, but I'm not using my board actively yet while waiting for the new wireless card to arrive. But I believe @Igor and @tkaiser used their boards enough and most likely didn't encounter anything similar. Just went through the dts between the solidrun repo: https://github.com/SolidRun/linux-stable/tree/linux-4.4.y-clearfog/arch/arm/boot/dts and I can't see any meaningful changes. Leaves the last part the image it boots and that's a much bigger needle to solve. Current Armbian default branch is based on SolidRun's 4.4.y-marvell branch with upstream patches, and next branch is mainline with some patches from 4.4.y (mostly support for the SFP) Really want to get it going with the current kernel stable so the system can be easily kept up to date in the future. Well, they are both stable. default branch is LTS 4.4.x, next branch is the latest stable release of the mainline kernel. Anyway, I thought you had some progress with freshly compiled Armbian images. Are there still any issues with them so you needed to go back to the Solid-Run images?
Valk Posted January 23, 2017 Author Posted January 23, 2017 Yeah I had improvements with the latest version in that the issue with the emmc throwing heeps of write errors seems to be fixed. Down side is now if I reboot the system it will no longer restart. I'm using the SolidRun image because I'm trying to get technical support from them. Unfortunately the official image gives me emmc write issues. The only one which works perfect is the 3.X kernel but that's both out of date and lacking kernel support for my 3G modem. By stable I only meant working reliably with the clearfog. Anyway hopefully the next batch of clearfogs arrive soon so I can test if it's these last 2 boards I have are faulty or there's an issue.
Valk Posted January 25, 2017 Author Posted January 25, 2017 Well got some feedback from SolidRun regarding the emmc issues. Apparently there's a firmware issue currently with the 4GB emmc (8GB is fine) which they're working to get a resolution. Hopefully when my next batch of boards with the 8GB emmc come the issue will be sorted. Really thanks for your help, while not an Armbian issue a lot of useful changes / fixes got done for the clearfog Oh and the resize code works fine now.
Pontus Jon Jensen Karlsson Posted March 31, 2017 Posted March 31, 2017 @Valk Can you attach the confirmation from SolidRun that this is in fact a firmware issue? It would be of great help since there's a few people trying to resolve the I/O timing issues on a driver level but have so far failed to do so.
Recommended Posts