Hadoop pseudo-distributed cluster construction
May 07, 2025 pm 04:45 PMSoftware preparation
I'm using a virtual machine with CentOS-6.6, with the host name repo. Refer to the steps to install a Linux virtual machine in Windows, I installed JDK in that virtual machine, refer to the guide to installing JDK in Linux. In addition, the virtual machine is configured with a key-free login itself, and the settings for configuring key-free login between each virtual machine are referenced. The download address of the Hadoop installation package is: http://miracleart.cn/link/8485694bae96aebc7c4fe6119599d0e0 .
-
Upload the Hadoop installation package to the server and decompress it
[root@repo ~]# tar zxvf hadoop-2.6.5.tar.gz -C /opt/apps/
-
Configure environment variables
# You can directly locate the last line of the file [root@repo hadoop-2.6.5]# vi /etc/profile export HADOOP_HOME=/opt/apps/hadoop-2.6.5 export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin [root@repo hadoop-2.6.5]# . /etc/profile
-
Modify the three configuration files of hadoop-env.sh, mapred-env.sh, and yarn-env.sh, and add JAVA_HOME
[root@repo hadoop]# pwd /opt/apps/hadoop-2.6.5/etc/hadoop [root@repo hadoop]# vi hadoop-env.sh export JAVA_HOME=/usr/local/jdk1.8.0_73 [root@repo hadoop]# vi mapred-env.sh export JAVA_HOME=/usr/local/jdk1.8.0_73 [root@repo hadoop]# vi yarn-env.sh export JAVA_HOME=/usr/local/jdk1.8.0_73
-
Modify the core-site.xml and hdfs-site.xml configuration files to configure pseudo-distributed related content
[root@repo hadoop]# vi core-site.xml <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://repo:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/var/hadoop/pseudo</value> </property> </configuration> [root@repo hadoop]# vi hdfs-site.xml <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.namenode.secondary.http-address</name> <value>repo:50090</value> </property> </configuration>
-
Modify the slaves configuration file and specify the node where the DataNode is located
[root@repo hadoop]# vi slaves repo
-
Format file system
[root@repo hadoop]# hadoop namenode --format # Success message17/09/16 21:17:11 INFO common.Storage: Storage directory /var/hadoop/pseudo/dfs/name has been successfully formatted.
-
Start HDFS and YARN
[root@repo hadoop]# start-dfs.sh Starting namesnodes on [repo] repo: starting namenode, logging to /opt/apps/hadoop-2.6.5/logs/hadoop-root-namenode-repo.out repo: starting datanode, logging to /opt/apps/hadoop-2.6.5/logs/hadoop-root-datanode-repo.out Starting secondary namesodes [repo] repo: starting secondarynamenode, logging to /opt/apps/hadoop-2.6.5/logs/hadoop-root-secondarynamenode-repo.out [root@repo hadoop]# start-yarn.sh starting yarn daemons starting resourcemanager, logging to /opt/hadoop-2.7.4/logs/yarn-root-resourcemanager-repo.out repo: starting nodemanager, logging to /opt/hadoop-2.7.4/logs/yarn-root-nodemanager-repo.out [root@repo hadoop]# jps 4368 Jps 3957 ResourceManager 3512 NameNode 3641 DataNode 4058 NodeManager 3805 SecondaryNameNode
Visit the WEB page
Build successfully!
The above is the detailed content of Hadoop pseudo-distributed cluster construction. For more information, please follow other related articles on the PHP Chinese website!

Hot AI Tools

Undress AI Tool
Undress images for free

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Clothoff.io
AI clothes remover

Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Hot Topics

When encountering the "Windowscan'taccesssharedfolderonnetwork", you can usually solve the problem through the following steps: 1. Turn on the network discovery and file sharing function and turn off password protection; 2. Make sure that the target computer is enabled to share and set the correct permissions; 3. Check the firewall rules and service status to ensure that it allows shared access; 4. Use the credential manager to add network credentials for long-term and stable connection.

When you encounter Windows stuck in the "GettingWindowsready, Don't turnoff your computer" interface, you should first confirm whether it is really stuck; 1. Observe whether the hard disk indicator light is flashing, 2. Check whether the fan sound has changed, 3. Wait at least 30 to 60 minutes to ensure that the system has enough time to complete the update operation.

When the Windows clipboard history is not working, you can check the following steps: 1. Confirm that the clipboard history function is enabled, the path is "Settings>System>Clipboard", and if it is not enabled, Win V will not respond; 2. Check whether the copy content type is limited, such as large images, special formats or file paths may not be saved; 3. Ensure that the system version supports it, Windows 101809 and above, and some enterprise versions or LTSC do not support it; 4. Try to restart the ClipboardUserService service or end the clipups.exe process; 5. Clear the clipboard cache or reset the settings, close and then turn on the "Clipboard History" or run the "echooff|clip" command to clean up the cache

To run programs as administrator, you can use Windows' own functions: 1. Right-click the menu to select "Run as administrator", which is suitable for temporary privilege hike scenarios; 2. Create a shortcut and check "Run as administrator" to achieve automatic privilege hike start; 3. Use the task scheduler to configure automated tasks, suitable for running programs that require permissions on a scheduled or background basis, pay attention to setting details such as path changes and permission checks.

Don't rush to reinstall the system when the computer is stuck in the Windows restart interface. You can try the following methods first: 1. Force shutdown and then restart. Apply to the situation where the update is stuck. Repeat two or three times or can skip the lag; 2. Enter the safe mode to check, select Start repair or system restore through troubleshooting. If you can enter safe mode, it may be a driver or software conflict; 3. Use the command prompt to repair the system files, enter the three commands sfc and dism in the recovery environment to repair the damaged files; 4. Check the recently installed hardware or driver, unplug the non-essential devices or uninstall the new driver to eliminate incompatibility issues. In most cases, the above steps can solve the phenomenon of restart lag. If it really doesn’t work, consider reinstalling the system and paying attention to backing up data in advance.

The error "This operation has been cancelled because of restrictions on the computer" is usually caused by permissions or policy restrictions. Solutions include: 1. Check whether to use an administrator account, and if not, switch or change the account type; 2. Run the program as an administrator, or set a shortcut to always run as an administrator; 3. Check Group Policy restrictions, set suspicious policies to "not configured" or "disabled", but be careful that there is no Group Policy Editor for the Home Edition; 4. If registry editing is disabled, you can re-enable it by creating a .reg file; 5. Troubleshoot third-party software interference, temporarily close the security software or management startup items. Trying the above methods in order usually solves the problem.

Corruption of user profile can be solved by creating a new account, using system tools to repair, manually migrating data, and modifying the registry. After confirming the problem, you can try the following methods in turn: 1. Create a new local account to determine whether it is the original account problem; 2. Use sfc/scannow and DISM tools to repair the system files; 3. Create a new user through the administrator account and manually migrate the old account files (including the hidden AppData directory); 4. Advanced users can try to locate the ProfileList item in the registry editor, check and repair the abnormal ProfileImagePath entry, and backup the registry before the operation.

The steps for Apache to modify the default port to 8080 are as follows: 1. Edit the Apache configuration file (such as /etc/apache2/ports.conf or /etc/httpd/conf/httpd.conf), and change Listen80 to Listen8080; 2. Modify the tag port in all virtual host configurations to 8080 to ensure that it is consistent with the listening port; 3. Check and open the support of the 8080 port by firewall (such as ufw and firewalld); 4. If SELinux or AppArmor is enabled, you need to set to allow Apache to use non-standard ports; 5. Restart the Apache service to make the configuration take effect; 6. Browser access
